diff --git a/.gitignore b/.gitignore index f738b40..1cbf25e 100644 --- a/.gitignore +++ b/.gitignore @@ -8,8 +8,11 @@ plans .claude* CLAUDE.md temp* -.next_features -.envv +!tempo +!tempo/ +!**/tempo.yaml +!**/tempo.yml +.env* .infisical.json .task* task* diff --git a/README.md b/README.md index 1c9a335..c13d6f9 100644 --- a/README.md +++ b/README.md @@ -50,8 +50,12 @@ Licensed under the [GNU Affero General Public License v3.0](LICENSE). Mistral) - Celery + Redis task queue: receipt extraction, statement parsing, and report delivery all run as durable background tasks that survive process restarts -- Prometheus metrics + Grafana dashboards for request latency, LLM latency, queue depth, - and rate-limit hits +- Full observability stack: Prometheus metrics + Alertmanager (Slack alerts) + Grafana + dashboards + Loki log aggregation + Grafana Tempo distributed tracing (OpenTelemetry + auto-instrumented: FastAPI, SQLAlchemy, HTTPX, Redis, Celery; manual LLM spans) +- Continuous Postgres backup via WAL-G to Cloudflare R2 (WAL archiving + daily base + backup, 7-day retention, point-in-time recovery) +- PgBouncer connection pooling (session mode) between application and Postgres - Demo mode: public Bruce Wayne account, automatically reset every 30 minutes via Celery Beat - GDPR Article 17 compliance: full account erasure and selective data deletion endpoints @@ -71,7 +75,9 @@ Licensed under the [GNU Affero General Public License v3.0](LICENSE). | Embeddings | Ollama (768-dim) + pgvector | | PDF generation | Puppeteer (headless Chromium via Next.js internal route) | | Email | Resend API | -| Observability | Prometheus + Grafana (provisioned dashboards) + Flower (Celery monitor) | +| Connection pooling | PgBouncer (session mode, scram-sha-256) | +| Observability | Prometheus + Alertmanager (Slack) + Grafana + Loki + Promtail + Tempo (OTel traces) + Flower (Celery monitor) + Sentry | +| Backup | WAL-G continuous backup to Cloudflare R2 (WAL archiving + daily base backup, 7-day retention, PITR) | | Secrets | Infisical (CLI-injected at runtime; no .env files in source) | | Containerization | Docker + Docker Compose | @@ -169,13 +175,19 @@ configured as `ADMIN_EMAIL`. That account is auto-approved. All other accounts s |---|---|---| | Frontend | http://localhost:3000 | Next.js app | | Backend API | http://localhost:8000 | FastAPI; `/docs` only when `DEBUG=true` | -| PostgreSQL | localhost:5432 | pgvector-enabled | +| PostgreSQL | localhost:5432 | pgvector-enabled; custom image with WAL-G in prod | +| PgBouncer | (internal only) | Connection pooler; prod only | | Redis | (internal only) | Task broker + cache; no host port exposed | | Celery worker | (no HTTP) | Processes receipt extraction and report tasks | | Celery Beat | (no HTTP) | Runs scheduled tasks (demo reset, recurring expenses) | +| WAL-G backup | (no HTTP) | Daily base backup sidecar; prod only | | Flower | http://localhost:5555 | Celery task monitor (dev only) | -| Prometheus | http://localhost:9090 | Metrics scraper | +| Prometheus | http://localhost:9090 | Metrics scraper (SSH tunnel in prod) | +| Alertmanager | http://localhost:9095 | Slack alert routing (SSH tunnel in prod) | | Grafana | http://localhost:3004 | Pre-provisioned dashboards; anonymous admin in dev | +| Loki | (internal only) | Log aggregation; queried via Grafana | +| Promtail | (internal only) | Docker log shipper to Loki | +| Tempo | (internal only) | Distributed trace storage (OTel OTLP HTTP) | --- @@ -268,16 +280,36 @@ Custom metrics exposed at `GET /metrics` (requires `Authorization: Bearer Loki: query logs across all containers with LogQL. +Grafana Explore -> Tempo: search traces by service, duration, or status. +Traces and logs are linked: a `traceID` field in structured logs becomes a clickable +link to the Tempo trace. + +In production, all monitoring services are bound to `127.0.0.1` only. Access via SSH tunnel: ```bash -ssh -L 9090:localhost:9090 -L 3004:localhost:3004 user@your-server +ssh -L 9090:localhost:9090 -L 9095:localhost:9095 -L 3004:localhost:3004 user@your-server ``` +### Alertmanager + +Alert rules are in `prometheus/alert_rules.yml` (`BackendDown`, `HighErrorRate`). +Alertmanager routes firing alerts to Slack via webhook. Set `SLACK_WEBHOOK_URL` in +Infisical. The webhook URL is never stored in config files - it is injected at container +startup via the entrypoint. + +### Distributed tracing (OpenTelemetry + Tempo) + +The backend and Celery worker auto-instrument FastAPI, SQLAlchemy, HTTPX, Redis, and +Celery via the OpenTelemetry SDK. Spans are exported to Grafana Tempo over OTLP HTTP. +Set `OTEL_ENDPOINT=http://tempo:4318` (already set in both compose files). Tracing is +disabled when `OTEL_ENDPOINT` is empty. + --- ## Demo mode @@ -359,30 +391,46 @@ The prod stack includes a `cloudflared` sidecar that establishes a Cloudflare Tu Set `CLOUDFLARE_TUNNEL_TOKEN` in Infisical and configure your Cloudflare dashboard to route your domain to `http://frontend:3000`. No port 443 needs to be open on the host. -### PostgreSQL backups +### PostgreSQL backups (WAL-G + Cloudflare R2) -A host-side systemd backup setup is in [`deploy/backup/`](deploy/backup/). Run once on -the production host: +Production uses WAL-G for continuous backup to Cloudflare R2. + +The production Postgres image (`postgres/Dockerfile`) starts with WAL archiving enabled: -```bash -sudo bash ./deploy/backup/install-spendhound-db-backup.sh ``` +-c wal_level=replica +-c archive_mode=on +-c archive_command=wal-g wal-push %p +-c archive_timeout=60 +``` + +Every WAL segment (up to 16 MB of changes, or at most 60 seconds of idle time) is +compressed and uploaded to R2. A separate sidecar (`wal-g-backup`) runs a daily base +backup at 02:00 UTC and prunes base backups older than 7 days. -This installs a systemd timer that: -- Runs daily at 03:15 UTC (`Persistent=true` catches missed runs) -- Dumps the `db` container with `pg_dump` -- Validates the archive with `pg_restore --list` and writes a SHA-256 checksum -- Atomically renames the validated archive into place -- Prunes old backups after the configured retention period -- Uses `set -Eeuo pipefail`, `umask 077`, and `flock` to prevent overlapping runs +Required Infisical secrets: `WALG_S3_PREFIX`, `WALG_R2_ENDPOINT`, `WALG_ACCESS_KEY_ID`, +`WALG_SECRET_ACCESS_KEY`. -Backups are stored in `/var/backups/spendhound`. +Trigger the first backup manually after initial deploy (before the 02:00 UTC cron fires): ```bash -sudo systemctl status spendhound-db-backup.timer --no-pager -sudo journalctl -u spendhound-db-backup.service -n 50 --no-pager +docker exec spendhound_walg_backup_prod /usr/local/bin/wal-g backup-push /var/lib/postgresql/data +docker exec spendhound_walg_backup_prod /usr/local/bin/wal-g backup-list ``` +To restore (PITR): + +```bash +# 1. Stop the stack and clear the data directory +# 2. Fetch the most recent base backup +docker exec spendhound_db_prod wal-g backup-fetch /var/lib/postgresql/data LATEST +# 3. Create recovery.signal and set recovery_target_time in postgresql.conf +# 4. Start Postgres - it replays WAL up to the target time +``` + +Maximum data loss is 60 seconds (set by `archive_timeout`). Retention: 7 daily base +backups + all WAL since the oldest retained base backup. + ### Single-worker constraint SpendHound is designed for `--workers 1`. The in-process `asyncio.Semaphore` that diff --git a/alertmanager/alertmanager.yml b/alertmanager/alertmanager.yml new file mode 100644 index 0000000..5336bcb --- /dev/null +++ b/alertmanager/alertmanager.yml @@ -0,0 +1,40 @@ +global: + resolve_timeout: 5m + +route: + group_by: ['alertname', 'severity'] + # Wait 30s after first alert fires before sending - collects related alerts into one message. + group_wait: 30s + # After the initial message, wait 5m before sending an update if new alerts join the group. + group_interval: 5m + # If an alert is still firing, re-notify every 4h so it doesn't silently stay broken. + repeat_interval: 4h + receiver: slack-alerts + +receivers: + - name: slack-alerts + slack_configs: + # Webhook URL is written to this file by the container entrypoint from $SLACK_WEBHOOK_URL. + # Same pattern as Prometheus reading the metrics bearer token from /run/metrics_token. + - api_url_file: /tmp/slack_webhook_url + channel: '#spendhound-alerts' + username: SpendHound + send_resolved: true + color: '{{ if eq .CommonLabels.severity "critical" }}danger{{ else }}warning{{ end }}' + title: '{{ if eq .Status "firing" }}[FIRING]{{ else }}[RESOLVED]{{ end }} {{ .CommonLabels.alertname }}' + text: | + {{ range .Alerts -}} + *{{ .Annotations.summary }}* + Severity: {{ .Labels.severity }} + {{ .Annotations.description }} + {{ if eq .Status "firing" }}Since: {{ .StartsAt.Format "2006-01-02 15:04 UTC" }}{{ else }}Resolved: {{ .EndsAt.Format "2006-01-02 15:04 UTC" }}{{ end }} + {{ end }} + +# Suppress downstream alerts when the root cause is already firing. +# If BackendDown fires, HighErrorRate is a symptom - no need for two messages. +inhibit_rules: + - source_match: + alertname: BackendDown + target_match: + alertname: HighErrorRate + equal: [] diff --git a/backend/app/celery_app.py b/backend/app/celery_app.py index a2748bc..15f8845 100644 --- a/backend/app/celery_app.py +++ b/backend/app/celery_app.py @@ -17,9 +17,31 @@ from celery import Celery from celery.schedules import crontab +from celery.signals import worker_init from app.config import settings +if settings.sentry_dsn: + import sentry_sdk + from sentry_sdk.integrations.celery import CeleryIntegration + + sentry_sdk.init( + dsn=settings.sentry_dsn, + environment=settings.sentry_environment, + traces_sample_rate=0.0, + send_default_pii=False, + integrations=[CeleryIntegration()], + ) + + +@worker_init.connect +def _on_worker_init(**kwargs: object) -> None: + if settings.otel_endpoint: + from app.services.tracing import setup_tracing + + setup_tracing(settings.otel_service_name, settings.otel_endpoint) + + celery_app = Celery( "spendhound", broker=settings.redis_url, @@ -51,11 +73,17 @@ ) # Beat schedule — runs when `celery beat` is started (see docker-compose celery_beat service). -# Wall-clock crontab (UTC :00 and :30) so the reset time is predictable and the -# frontend can derive an accurate countdown without any server-side state. celery_app.conf.beat_schedule = { + # Wall-clock crontab (UTC :00 and :30) so the reset time is predictable and the + # frontend can derive an accurate countdown without any server-side state. "reset-demo-user-on-the-half-hour": { "task": "app.tasks.demo_tasks.reset_demo_user", "schedule": crontab(minute="0,30"), }, + # 1st of each month at 00:20 UTC - matches the former systemd timer schedule. + # Fans out one deliver_monthly_report task per eligible user. + "dispatch-monthly-reports": { + "task": "app.tasks.report_tasks.dispatch_monthly_reports", + "schedule": crontab(hour=0, minute=20, day_of_month=1), + }, } diff --git a/backend/app/config.py b/backend/app/config.py index 86b03d1..be010b6 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -152,6 +152,22 @@ class Settings(BaseSettings): default="", description="Bearer token required to scrape /metrics. Empty = endpoint blocked. Set in Infisical for prod.", ) + sentry_dsn: str = Field( + default="", + description="Sentry DSN for error tracking. Empty = Sentry disabled. Set in Infisical for prod.", + ) + sentry_environment: str = Field( + default="production", + description="Sentry environment tag sent with every event (production / development).", + ) + otel_endpoint: str = Field( + default="", + description="OTLP HTTP base URL for OpenTelemetry traces (e.g. http://tempo:4318). Empty = tracing disabled.", + ) + otel_service_name: str = Field( + default="spendhound", + description="OTel service name shown in Grafana Tempo. Set per container: spendhound-api, spendhound-worker.", + ) debug: bool = Field(default=False, description="Enable debug mode") cors_origins: list[str] = Field( diff --git a/backend/app/main.py b/backend/app/main.py index 984f96b..a65cc8e 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -21,6 +21,28 @@ from app.services.cache import close_redis, get_celery_queue_depth, init_redis from app.services.metrics import RATE_LIMIT_HITS_TOTAL, RECEIPT_QUEUE_DEPTH, classify_limit_type +if settings.sentry_dsn: + import sentry_sdk + from sentry_sdk.integrations.fastapi import FastApiIntegration + from sentry_sdk.integrations.starlette import StarletteIntegration + + sentry_sdk.init( + dsn=settings.sentry_dsn, + environment=settings.sentry_environment, + traces_sample_rate=0.0, + send_default_pii=False, + integrations=[StarletteIntegration(), FastApiIntegration()], + ) + +if settings.otel_endpoint: + from app.services.tracing import setup_tracing + + setup_tracing(settings.otel_service_name, settings.otel_endpoint) + # FastAPI instrumentation is deferred to create_app() via instrument_fastapi() + # because the FastAPI app must exist before middleware can be attached. + # Calling FastAPIInstrumentor().instrument() here (before app creation) patches + # FastAPI.__init__ globally and conflicts with the Prometheus Instrumentator. + logger = structlog.get_logger(__name__) @@ -85,6 +107,11 @@ async def _rate_limit_handler(request: Request, exc: Exception) -> Response: app.add_exception_handler(RateLimitExceeded, _rate_limit_handler) + if settings.otel_endpoint: + from app.services.tracing import instrument_fastapi + + instrument_fastapi(app) + app.add_middleware( CORSMiddleware, allow_origins=settings.cors_origins, @@ -93,6 +120,27 @@ async def _rate_limit_handler(request: Request, exc: Exception) -> Response: allow_headers=["*"], ) + if settings.sentry_dsn: + + @app.middleware("http") + async def _attach_sentry_user(request: Request, call_next): # type: ignore[no-untyped-def] + auth = request.headers.get("authorization", "") + if auth.startswith("Bearer "): + try: + from jose import jwt as _jwt + + payload = _jwt.decode( + auth.removeprefix("Bearer "), + settings.jwt_secret, + algorithms=[settings.jwt_algorithm], + ) + user_id = payload.get("sub") + if user_id: + sentry_sdk.set_user({"id": user_id}) + except Exception: + pass + return await call_next(request) + # Starlette 0.40+ adds _IncludedRouter objects to app.routes when include_router() # is used. prometheus-fastapi-instrumentator unconditionally accesses route.path on # every route, but _IncludedRouter has no .path → AttributeError on every request. diff --git a/backend/app/services/metrics.py b/backend/app/services/metrics.py index 9939312..88b863d 100644 --- a/backend/app/services/metrics.py +++ b/backend/app/services/metrics.py @@ -17,6 +17,7 @@ import time from collections.abc import AsyncGenerator +from opentelemetry import trace as _otel_trace from prometheus_client import Counter, Gauge, Histogram from app.services.llm.base import BaseLLMProvider, LLMConfig, Message @@ -58,13 +59,22 @@ def __init__(self, inner: BaseLLMProvider, provider_name: str) -> None: self._provider_name = provider_name async def complete(self, messages: list[Message], config: LLMConfig | None = None) -> str: - start = time.monotonic() - try: - return await self._inner.complete(messages, config) - finally: - LLM_RESPONSE_SECONDS.labels(provider=self._provider_name).observe( - time.monotonic() - start - ) + tracer = _otel_trace.get_tracer(__name__) + with tracer.start_as_current_span( + "llm.complete", + attributes={"gen_ai.system": self._provider_name}, + ) as span: + start = time.monotonic() + try: + return await self._inner.complete(messages, config) + except Exception as exc: + span.record_exception(exc) + span.set_status(_otel_trace.Status(_otel_trace.StatusCode.ERROR)) + raise + finally: + LLM_RESPONSE_SECONDS.labels(provider=self._provider_name).observe( + time.monotonic() - start + ) async def stream( self, messages: list[Message], config: LLMConfig | None = None diff --git a/backend/app/services/tracing.py b/backend/app/services/tracing.py new file mode 100644 index 0000000..4ca0dd2 --- /dev/null +++ b/backend/app/services/tracing.py @@ -0,0 +1,73 @@ +"""OpenTelemetry tracing setup for SpendHound. + +Call setup_tracing() once at process startup. No-op when otel_endpoint is empty. +Idempotent - safe to call multiple times; only the first call takes effect. + +Instrumented libraries (auto): + FastAPI/Starlette - HTTP request spans with route, method, status + SQLAlchemy - every DB query as a child span + HTTPX - outbound HTTP calls (LLM provider APIs, Puppeteer) + Redis - cache get/set/delete spans + Celery - task publish (in API process) + task execute (in worker) + +Manual spans are added in MeteredLLMProvider.complete() with provider attributes. +""" + +from __future__ import annotations + +import structlog + +logger = structlog.get_logger(__name__) + +_initialized = False + + +def setup_tracing(service_name: str, endpoint: str) -> None: + """Configure the global TracerProvider and auto-instrument common libraries. + + service_name: label in Grafana Tempo service map (e.g. spendhound-api) + endpoint: OTLP HTTP base URL for Grafana Tempo (e.g. http://tempo:4318) + """ + global _initialized + if _initialized or not endpoint: + return + _initialized = True + + from opentelemetry import trace + from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter + from opentelemetry.instrumentation.celery import CeleryInstrumentor + from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor + from opentelemetry.instrumentation.redis import RedisInstrumentor + from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor + from opentelemetry.sdk.resources import SERVICE_NAME, Resource + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import BatchSpanProcessor + + resource = Resource(attributes={SERVICE_NAME: service_name}) + provider = TracerProvider(resource=resource) + provider.add_span_processor( + BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{endpoint}/v1/traces")) + ) + trace.set_tracer_provider(provider) + + SQLAlchemyInstrumentor().instrument() + HTTPXClientInstrumentor().instrument() + RedisInstrumentor().instrument() + CeleryInstrumentor().instrument() # type: ignore[no-untyped-call] + + logger.info("tracing.initialized", service=service_name, endpoint=endpoint) + + +def instrument_fastapi(app: object) -> None: + """Add OTel HTTP tracing middleware to an already-created FastAPI app. + + Must be called after setup_tracing() and after the FastAPI app instance + exists. Calling FastAPIInstrumentor().instrument() without an app patches + FastAPI.__init__ globally and conflicts with the Prometheus Instrumentator + middleware added later in create_app(), producing an unpredictable + middleware stack that causes intermittent request failures. + """ + from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor + + FastAPIInstrumentor.instrument_app(app) # type: ignore[arg-type] + logger.info("tracing.fastapi_instrumented") diff --git a/backend/pyproject.toml b/backend/pyproject.toml index bb8c63b..9da70ce 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -45,6 +45,15 @@ dependencies = [ "celery[redis]>=5.3", "prometheus-client>=0.20.0", "prometheus-fastapi-instrumentator==7.1.0", + "sentry-sdk[fastapi]>=2.0.0", + # OpenTelemetry - distributed tracing to Grafana Tempo + "opentelemetry-sdk>=1.24.0", + "opentelemetry-exporter-otlp-proto-http>=1.24.0", + "opentelemetry-instrumentation-fastapi>=0.45b0", + "opentelemetry-instrumentation-sqlalchemy>=0.45b0", + "opentelemetry-instrumentation-httpx>=0.45b0", + "opentelemetry-instrumentation-redis>=0.45b0", + "opentelemetry-instrumentation-celery>=0.45b0", ] [project.optional-dependencies] diff --git a/deploy/reports/install-spendhound-monthly-reports.sh b/deploy/reports/install-spendhound-monthly-reports.sh deleted file mode 100644 index c1c5a28..0000000 --- a/deploy/reports/install-spendhound-monthly-reports.sh +++ /dev/null @@ -1,107 +0,0 @@ -#!/usr/bin/env bash - -set -Eeuo pipefail - -umask 022 - -SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" -REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../.." && pwd -P)" - -RUNNER_SOURCE="${SCRIPT_DIR}/spendhound-monthly-reports.sh" -SERVICE_SOURCE="${SCRIPT_DIR}/spendhound-monthly-reports.service" -TIMER_SOURCE="${SCRIPT_DIR}/spendhound-monthly-reports.timer" -COMPOSE_FILE="${REPO_ROOT}/docker-compose.prod.yml" - -SYSTEMD_DIR="/etc/systemd/system" -SERVICE_NAME="spendhound-monthly-reports.service" -TIMER_NAME="spendhound-monthly-reports.timer" -OVERRIDE_DIR="${SYSTEMD_DIR}/${SERVICE_NAME}.d" -OVERRIDE_FILE="${OVERRIDE_DIR}/override.conf" -SERVICE_DEST="${SYSTEMD_DIR}/${SERVICE_NAME}" -TIMER_DEST="${SYSTEMD_DIR}/${TIMER_NAME}" - -log() { - printf '[%s] %s\n' "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" "$*" -} - -fail() { - log "ERROR: $*" - exit 1 -} - -require_command() { - command -v "$1" >/dev/null 2>&1 || fail "Required command not found: $1" -} - -require_root() { - [[ "${EUID}" -eq 0 ]] || fail "Run this installer as root (for example: sudo bash ./deploy/reports/install-spendhound-monthly-reports.sh)" -} - -write_override() { - local tmp_file - tmp_file="$(mktemp)" - trap 'rm -f -- "${tmp_file}"' RETURN - - cat >"${tmp_file}" </dev/null 2>&1; then - fail "systemd does not appear to be available on this host" -fi - -[[ -f "${RUNNER_SOURCE}" ]] || fail "Runner script not found at ${RUNNER_SOURCE}" -[[ -f "${SERVICE_SOURCE}" ]] || fail "Service unit not found at ${SERVICE_SOURCE}" -[[ -f "${TIMER_SOURCE}" ]] || fail "Timer unit not found at ${TIMER_SOURCE}" -[[ -f "${COMPOSE_FILE}" ]] || fail "Compose file not found at ${COMPOSE_FILE}" - -log "Installing SpendHound monthly reports systemd units" - -install -d -m 0755 "${SYSTEMD_DIR}" -chmod 0755 "${RUNNER_SOURCE}" - -install -m 0644 "${SERVICE_SOURCE}" "${SERVICE_DEST}" -install -m 0644 "${TIMER_SOURCE}" "${TIMER_DEST}" -write_override - -systemctl daemon-reload -systemctl enable "${TIMER_NAME}" >/dev/null - -if systemctl is-active --quiet "${TIMER_NAME}"; then - systemctl restart "${TIMER_NAME}" -else - systemctl start "${TIMER_NAME}" -fi - -log "Monthly reports installer completed successfully" -printf '\n' -printf 'Installed unit files:\n' -printf ' - %s\n' "${SERVICE_DEST}" -printf ' - %s\n' "${TIMER_DEST}" -printf 'Installed override:\n' -printf ' - %s\n' "${OVERRIDE_FILE}" -printf '\n' -printf 'Verification commands:\n' -printf ' - systemctl status %s --no-pager\n' "${TIMER_NAME}" -printf ' - systemctl cat %s\n' "${SERVICE_NAME}" -printf ' - systemctl list-timers %s\n' "${TIMER_NAME}" -printf ' - journalctl -u %s -n 50 --no-pager\n' "${SERVICE_NAME}" -printf ' - systemctl start %s\n' "${SERVICE_NAME}" diff --git a/deploy/reports/spendhound-monthly-reports.service b/deploy/reports/spendhound-monthly-reports.service deleted file mode 100644 index 45d9d21..0000000 --- a/deploy/reports/spendhound-monthly-reports.service +++ /dev/null @@ -1,17 +0,0 @@ -[Unit] -Description=SpendHound monthly reports job -Documentation=https://github.com/sumdher/spendhound -Wants=docker.service -After=docker.service - -[Service] -Type=oneshot -User=root -Group=root -WorkingDirectory=/opt/spendhound -ExecStart=/usr/bin/env bash /opt/spendhound/deploy/reports/spendhound-monthly-reports.sh - -# Optional overrides for non-default deployments can be added here or via a drop-in: -# Environment=SPENDHOUND_REPORTS_COMPOSE_FILE=/opt/spendhound/docker-compose.prod.yml -# Environment=SPENDHOUND_REPORTS_PROJECT_DIR=/opt/spendhound -# Environment=SPENDHOUND_REPORTS_SERVICE_NAME=backend diff --git a/deploy/reports/spendhound-monthly-reports.sh b/deploy/reports/spendhound-monthly-reports.sh deleted file mode 100644 index ac4187a..0000000 --- a/deploy/reports/spendhound-monthly-reports.sh +++ /dev/null @@ -1,54 +0,0 @@ -#!/usr/bin/env bash - -set -Eeuo pipefail - -SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../.." && pwd)" - -COMPOSE_FILE="${SPENDHOUND_REPORTS_COMPOSE_FILE:-${REPO_ROOT}/docker-compose.prod.yml}" -COMPOSE_PROJECT_DIR="${SPENDHOUND_REPORTS_PROJECT_DIR:-${REPO_ROOT}}" -SERVICE_NAME="${SPENDHOUND_REPORTS_SERVICE_NAME:-backend}" -LOCK_FILE="${SPENDHOUND_REPORTS_LOCK_FILE:-/var/lock/spendhound-monthly-reports.lock}" - -log() { - printf '%s %s\n' "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" "$*" -} - -fail() { - log "ERROR: $*" - exit 1 -} - -require_command() { - command -v "$1" >/dev/null 2>&1 || fail "Required command not found: $1" -} - -require_command docker -require_command flock -require_command install - -[[ -f "${COMPOSE_FILE}" ]] || fail "Compose file not found at ${COMPOSE_FILE}" - -install -d -m 0700 "$(dirname -- "${LOCK_FILE}")" - -exec 9>"${LOCK_FILE}" -if ! flock -n 9; then - fail "Another monthly report job is already running" -fi - -DOCKER_COMPOSE=(docker compose --project-directory "${COMPOSE_PROJECT_DIR}" -f "${COMPOSE_FILE}") - -BACKEND_CONTAINER_ID="$(${DOCKER_COMPOSE[@]} ps -q "${SERVICE_NAME}" 2>/dev/null || true)" -[[ -n "${BACKEND_CONTAINER_ID}" ]] || fail "Backend service '${SERVICE_NAME}' is not running" - -BACKEND_RUNNING="$(docker inspect -f '{{.State.Running}}' "${BACKEND_CONTAINER_ID}" 2>/dev/null || true)" -[[ "${BACKEND_RUNNING}" == "true" ]] || fail "Backend container '${BACKEND_CONTAINER_ID}' is not running" - -BACKEND_HEALTH="$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "${BACKEND_CONTAINER_ID}" 2>/dev/null || true)" -if [[ "${BACKEND_HEALTH}" != "healthy" && "${BACKEND_HEALTH}" != "none" ]]; then - fail "Backend container health is '${BACKEND_HEALTH}', refusing job execution" -fi - -log "Starting SpendHound monthly reports job via backend service '${SERVICE_NAME}'" -"${DOCKER_COMPOSE[@]}" exec -T "${SERVICE_NAME}" python -m app.jobs.monthly_reports -log "Monthly reports job finished" diff --git a/deploy/reports/spendhound-monthly-reports.timer b/deploy/reports/spendhound-monthly-reports.timer deleted file mode 100644 index cd44520..0000000 --- a/deploy/reports/spendhound-monthly-reports.timer +++ /dev/null @@ -1,11 +0,0 @@ -[Unit] -Description=Run SpendHound monthly reports on the first day of each month - -[Timer] -OnCalendar=*-*-01 00:20:00 -RandomizedDelaySec=20m -Persistent=true -Unit=spendhound-monthly-reports.service - -[Install] -WantedBy=timers.target diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 8fdcb8c..26df8c3 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -3,14 +3,39 @@ services: db: - # Pin by digest so a tag re-push cannot silently swap the image. - # To update: docker pull pgvector/pgvector:pg16 && docker inspect --format='{{index .RepoDigests 0}}' pgvector/pgvector:pg16 - image: pgvector/pgvector@sha256:7d400e340efb42f4d8c9c12c6427adb253f726881a9985d2a471bf0eed824dff + # Custom image: pgvector:pg16 + WAL-G binary (see postgres/Dockerfile). + # The base image digest is pinned inside the Dockerfile FROM line. + build: + context: ./postgres + dockerfile: Dockerfile container_name: spendhound_db_prod + # Enable WAL archiving. archive_mode requires a server restart to toggle, + # so it must be set here (not via ALTER SYSTEM). archive_timeout=60 caps + # WAL data loss at ~60 seconds when the database is idle (no writes fill + # a full 16 MB segment naturally). + command: + - postgres + - -c + - wal_level=replica + - -c + - archive_mode=on + - -c + - archive_command=wal-g wal-push %p + - -c + - archive_timeout=60 environment: POSTGRES_USER: ${POSTGRES_USER:-spendhound} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} POSTGRES_DB: ${POSTGRES_DB:-spendhound} + # WAL-G credentials - injected by infisical run. + # archive_command inherits the Postgres process environment, so wal-g + # picks these up automatically on each WAL segment upload. + WALG_S3_PREFIX: ${WALG_S3_PREFIX} + AWS_ENDPOINT: ${WALG_R2_ENDPOINT} + AWS_ACCESS_KEY_ID: ${WALG_ACCESS_KEY_ID} + AWS_SECRET_ACCESS_KEY: ${WALG_SECRET_ACCESS_KEY} + AWS_REGION: auto + WALG_S3_USE_PATH_STYLE: "true" volumes: - postgres_data:/var/lib/postgresql/data healthcheck: @@ -27,6 +52,47 @@ services: reservations: memory: 256M + wal-g-backup: + # Shares the same image as db (WAL-G + supercronic already installed). + # Runs the daily base backup cron. WAL segment archiving is handled by + # Postgres itself via archive_command - this sidecar only does base backups. + # On FIRST DEPLOYMENT run a manual base backup before this schedule fires: + # docker exec spendhound_walg_backup_prod wal-g backup-push "$PGDATA" + build: + context: ./postgres + dockerfile: Dockerfile + container_name: spendhound_walg_backup_prod + entrypoint: [] + command: [/usr/local/bin/supercronic, /etc/backup-cron] + # Run as the postgres user (UID 999) so it can read the data directory. + user: "999:999" + environment: + PGDATA: /var/lib/postgresql/data + PGHOST: db + PGPORT: "5432" + PGUSER: ${POSTGRES_USER:-spendhound} + PGPASSWORD: ${POSTGRES_PASSWORD} + PGDATABASE: ${POSTGRES_DB:-spendhound} + WALG_S3_PREFIX: ${WALG_S3_PREFIX} + AWS_ENDPOINT: ${WALG_R2_ENDPOINT} + AWS_ACCESS_KEY_ID: ${WALG_ACCESS_KEY_ID} + AWS_SECRET_ACCESS_KEY: ${WALG_SECRET_ACCESS_KEY} + AWS_REGION: auto + WALG_S3_USE_PATH_STYLE: "true" + volumes: + - postgres_data:/var/lib/postgresql/data + - ./postgres/backup-cron:/etc/backup-cron:ro + depends_on: + db: + condition: service_healthy + security_opt: + - no-new-privileges:true + restart: unless-stopped + deploy: + resources: + limits: + memory: 256M + redis: image: redis:7-alpine container_name: spendhound_redis_prod @@ -49,6 +115,39 @@ services: reservations: memory: 64M + pgbouncer: + image: edoburu/pgbouncer:v1.23.1-p3 + container_name: spendhound_pgbouncer_prod + # Write userlist.txt at startup from env vars - same pattern as Prometheus + # writing the metrics token to /run/metrics_token. + entrypoint: + - /bin/sh + - -c + - | + printf '"%s" "%s"\n' "$$POSTGRES_USER" "$$POSTGRES_PASSWORD" > /tmp/pgbouncer_userlist.txt + exec pgbouncer /etc/pgbouncer/pgbouncer.ini + environment: + POSTGRES_USER: ${POSTGRES_USER:-spendhound} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + volumes: + - ./pgbouncer/pgbouncer.ini:/etc/pgbouncer/pgbouncer.ini:ro + depends_on: + db: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "nc -z 127.0.0.1 5432"] + interval: 5s + timeout: 3s + retries: 5 + start_period: 5s + security_opt: + - no-new-privileges:true + restart: unless-stopped + deploy: + resources: + limits: + memory: 64M + backend: build: context: ./backend @@ -59,7 +158,7 @@ services: - path: ./backend/.env # non-secret config only — optional, all secrets via env block required: false environment: - DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-spendhound}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB:-spendhound} + DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-spendhound}:${POSTGRES_PASSWORD}@pgbouncer:5432/${POSTGRES_DB:-spendhound}?prepared_statement_cache_size=0 REDIS_URL: ${REDIS_URL:-redis://redis:6379/0} # Secrets — injected by `infisical run` via Makefile GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID} @@ -74,6 +173,9 @@ services: MONTHLY_REPORTS_FRONTEND_TOKEN: ${MONTHLY_REPORTS_FRONTEND_TOKEN} MONTHLY_REPORTS_FRONTEND_TOKEN_HEADER: ${MONTHLY_REPORTS_FRONTEND_TOKEN_HEADER:-X-SpendHound-Internal-Token} METRICS_TOKEN: ${METRICS_TOKEN} + SENTRY_DSN: ${SENTRY_DSN:-} + OTEL_ENDPOINT: http://tempo:4318 + OTEL_SERVICE_NAME: spendhound-api volumes: - receipt_storage:/app/storage/receipts command: @@ -88,6 +190,8 @@ services: condition: service_healthy redis: condition: service_healthy + pgbouncer: + condition: service_healthy security_opt: - no-new-privileges:true cap_drop: @@ -118,7 +222,7 @@ services: - path: ./backend/.env required: false environment: - DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-spendhound}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB:-spendhound} + DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-spendhound}:${POSTGRES_PASSWORD}@pgbouncer:5432/${POSTGRES_DB:-spendhound}?prepared_statement_cache_size=0 REDIS_URL: ${REDIS_URL:-redis://redis:6379/0} # Secrets — injected by `infisical run` via Makefile GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID} @@ -132,6 +236,9 @@ services: LLM_KEY_ENCRYPTION_SECRET: ${LLM_KEY_ENCRYPTION_SECRET} MONTHLY_REPORTS_FRONTEND_TOKEN: ${MONTHLY_REPORTS_FRONTEND_TOKEN} MONTHLY_REPORTS_FRONTEND_TOKEN_HEADER: ${MONTHLY_REPORTS_FRONTEND_TOKEN_HEADER:-X-SpendHound-Internal-Token} + SENTRY_DSN: ${SENTRY_DSN:-} + OTEL_ENDPOINT: http://tempo:4318 + OTEL_SERVICE_NAME: spendhound-worker command: celery -A app.celery_app worker --loglevel=info --concurrency=1 volumes: - receipt_storage:/app/storage/receipts @@ -140,6 +247,8 @@ services: condition: service_healthy redis: condition: service_healthy + pgbouncer: + condition: service_healthy security_opt: - no-new-privileges:true cap_drop: @@ -154,6 +263,39 @@ services: extra_hosts: - "host.docker.internal:host-gateway" + celery_beat: + build: + context: ./backend + dockerfile: Dockerfile + target: production + container_name: spendhound_celery_beat_prod + env_file: + - path: ./backend/.env + required: false + environment: + DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-spendhound}:${POSTGRES_PASSWORD}@pgbouncer:5432/${POSTGRES_DB:-spendhound}?prepared_statement_cache_size=0 + REDIS_URL: ${REDIS_URL:-redis://redis:6379/0} + GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID} + GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET} + JWT_SECRET: ${JWT_SECRET} + LLM_KEY_ENCRYPTION_SECRET: ${LLM_KEY_ENCRYPTION_SECRET} + SENTRY_DSN: ${SENTRY_DSN:-} + command: celery -A app.celery_app beat --loglevel=info --schedule=/tmp/celerybeat-schedule + depends_on: + db: + condition: service_healthy + redis: + condition: service_healthy + security_opt: + - no-new-privileges:true + cap_drop: + - ALL + restart: unless-stopped + deploy: + resources: + limits: + memory: 256M + prometheus: image: prom/prometheus:v2.53.0 container_name: spendhound_prometheus_prod @@ -163,7 +305,7 @@ services: - /bin/sh - -c - | - printf '%s' "$$METRICS_TOKEN" > /run/metrics_token + printf '%s' "$$METRICS_TOKEN" > /tmp/metrics_token exec /bin/prometheus \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/prometheus \ @@ -174,6 +316,7 @@ services: - "127.0.0.1:9090:9090" # SSH tunnel: ssh -L 9090:localhost:9090 user@server volumes: - ./prometheus.prod.yml:/etc/prometheus/prometheus.yml:ro + - ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules.yml:ro - prometheus_data:/prometheus depends_on: backend: @@ -209,6 +352,80 @@ services: limits: memory: 256M + tempo: + image: grafana/tempo:2.6.0 + container_name: spendhound_tempo_prod + command: -config.file=/etc/tempo/tempo.yml + volumes: + - ./tempo/tempo.yml:/etc/tempo/tempo.yml:ro + - tempo_data:/var/tempo + security_opt: + - no-new-privileges:true + restart: unless-stopped + deploy: + resources: + limits: + memory: 512M + + loki: + image: grafana/loki:3.1.0 + container_name: spendhound_loki_prod + command: -config.file=/etc/loki/loki.yml + volumes: + - ./loki/loki.yml:/etc/loki/loki.yml:ro + - loki_data:/loki + security_opt: + - no-new-privileges:true + restart: unless-stopped + deploy: + resources: + limits: + memory: 512M + + promtail: + image: grafana/promtail:3.1.0 + container_name: spendhound_promtail_prod + command: -config.file=/etc/promtail/promtail.yml + volumes: + - ./promtail/promtail.yml:/etc/promtail/promtail.yml:ro + - /var/run/docker.sock:/var/run/docker.sock:ro + - /var/lib/docker/containers:/var/lib/docker/containers:ro + - promtail_positions:/var/promtail + depends_on: + - loki + restart: unless-stopped + + alertmanager: + image: prom/alertmanager:v0.27.0 + container_name: spendhound_alertmanager_prod + # Write SLACK_WEBHOOK_URL to a file; alertmanager.yml reads it via api_url_file. + # Same pattern as Prometheus reading the metrics token from /run/metrics_token. + entrypoint: + - /bin/sh + - -c + - | + printf '%s' "$$SLACK_WEBHOOK_URL" > /tmp/slack_webhook_url + exec /bin/alertmanager \ + --config.file=/etc/alertmanager/alertmanager.yml \ + --storage.path=/alertmanager \ + --web.listen-address=0.0.0.0:9093 + environment: + SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL} + ports: + - "127.0.0.1:9095:9093" # SSH tunnel: ssh -L 9095:localhost:9095 user@server + volumes: + - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro + - alertmanager_data:/alertmanager + depends_on: + - prometheus + security_opt: + - no-new-privileges:true + restart: unless-stopped + deploy: + resources: + limits: + memory: 128M + frontend: build: context: ./frontend @@ -273,4 +490,8 @@ volumes: postgres_data: receipt_storage: prometheus_data: + alertmanager_data: + loki_data: + promtail_positions: + tempo_data: grafana_data: diff --git a/docker-compose.yml b/docker-compose.yml index 6584225..db8c6ff 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -31,6 +31,31 @@ services: retries: 5 restart: unless-stopped + pgbouncer: + image: edoburu/pgbouncer:v1.23.1-p3 + container_name: spendhound_pgbouncer + entrypoint: + - /bin/sh + - -c + - | + printf '"%s" "%s"\n' "$$POSTGRES_USER" "$$POSTGRES_PASSWORD" > /tmp/pgbouncer_userlist.txt + exec pgbouncer /etc/pgbouncer/pgbouncer.ini + environment: + POSTGRES_USER: spendhound + POSTGRES_PASSWORD: localdev + volumes: + - ./pgbouncer/pgbouncer.ini:/etc/pgbouncer/pgbouncer.ini:ro + depends_on: + db: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "nc -z 127.0.0.1 5432"] + interval: 5s + timeout: 3s + retries: 5 + start_period: 5s + restart: unless-stopped + backend: build: context: ./backend @@ -41,7 +66,7 @@ services: - path: ./backend/.env # non-secret config only — optional, all secrets via env block required: false environment: - DATABASE_URL: postgresql+asyncpg://spendhound:localdev@db:5432/spendhound + DATABASE_URL: postgresql+asyncpg://spendhound:localdev@pgbouncer:5432/spendhound?prepared_statement_cache_size=0 REDIS_URL: redis://redis:6379/0 # Secrets — injected by `infisical run` via Makefile GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID} @@ -56,6 +81,9 @@ services: MONTHLY_REPORTS_FRONTEND_TOKEN: ${MONTHLY_REPORTS_FRONTEND_TOKEN} MONTHLY_REPORTS_FRONTEND_TOKEN_HEADER: ${MONTHLY_REPORTS_FRONTEND_TOKEN_HEADER:-X-SpendHound-Internal-Token} METRICS_TOKEN: dev-metrics-token + SENTRY_DSN: ${SENTRY_DSN:-} + OTEL_ENDPOINT: http://tempo:4318 + OTEL_SERVICE_NAME: spendhound-api ports: - "8000:8000" volumes: @@ -69,6 +97,8 @@ services: condition: service_healthy redis: condition: service_healthy + pgbouncer: + condition: service_healthy healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 30s @@ -88,7 +118,7 @@ services: - path: ./backend/.env required: false environment: - DATABASE_URL: postgresql+asyncpg://spendhound:localdev@db:5432/spendhound + DATABASE_URL: postgresql+asyncpg://spendhound:localdev@pgbouncer:5432/spendhound?prepared_statement_cache_size=0 REDIS_URL: redis://redis:6379/0 GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID} GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET} @@ -101,6 +131,9 @@ services: LLM_KEY_ENCRYPTION_SECRET: ${LLM_KEY_ENCRYPTION_SECRET} MONTHLY_REPORTS_FRONTEND_TOKEN: ${MONTHLY_REPORTS_FRONTEND_TOKEN} MONTHLY_REPORTS_FRONTEND_TOKEN_HEADER: ${MONTHLY_REPORTS_FRONTEND_TOKEN_HEADER:-X-SpendHound-Internal-Token} + SENTRY_DSN: ${SENTRY_DSN:-} + OTEL_ENDPOINT: http://tempo:4318 + OTEL_SERVICE_NAME: spendhound-worker command: celery -A app.celery_app worker --loglevel=info --concurrency=1 volumes: - ./backend:/app @@ -110,6 +143,8 @@ services: condition: service_healthy redis: condition: service_healthy + pgbouncer: + condition: service_healthy restart: unless-stopped extra_hosts: - "host.docker.internal:host-gateway" @@ -124,7 +159,7 @@ services: - path: ./backend/.env required: false environment: - DATABASE_URL: postgresql+asyncpg://spendhound:localdev@db:5432/spendhound + DATABASE_URL: postgresql+asyncpg://spendhound:localdev@pgbouncer:5432/spendhound?prepared_statement_cache_size=0 REDIS_URL: redis://redis:6379/0 GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID} GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET} @@ -132,6 +167,7 @@ services: ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-} OPENAI_API_KEY: ${OPENAI_API_KEY:-} LLM_KEY_ENCRYPTION_SECRET: ${LLM_KEY_ENCRYPTION_SECRET} + SENTRY_DSN: ${SENTRY_DSN:-} command: celery -A app.celery_app beat --loglevel=info --schedule=/tmp/celerybeat-schedule volumes: - ./backend:/app @@ -155,11 +191,35 @@ services: - "127.0.0.1:9090:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro + - ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules.yml:ro - prometheus_data:/prometheus depends_on: - backend restart: unless-stopped + alertmanager: + image: prom/alertmanager:v0.27.0 + container_name: spendhound_alertmanager + entrypoint: + - /bin/sh + - -c + - | + printf '%s' "$$SLACK_WEBHOOK_URL" > /tmp/slack_webhook_url + exec /bin/alertmanager \ + --config.file=/etc/alertmanager/alertmanager.yml \ + --storage.path=/alertmanager \ + --web.listen-address=0.0.0.0:9093 + environment: + SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL:-} + ports: + - "127.0.0.1:9095:9093" + volumes: + - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro + - alertmanager_data:/alertmanager + depends_on: + - prometheus + restart: unless-stopped + grafana: image: grafana/grafana:11.2.0 container_name: spendhound_grafana @@ -176,6 +236,37 @@ services: - prometheus restart: unless-stopped + loki: + image: grafana/loki:3.1.0 + container_name: spendhound_loki + command: -config.file=/etc/loki/loki.yml + volumes: + - ./loki/loki.yml:/etc/loki/loki.yml:ro + - loki_data:/loki + restart: unless-stopped + + promtail: + image: grafana/promtail:3.1.0 + container_name: spendhound_promtail + command: -config.file=/etc/promtail/promtail.yml + volumes: + - ./promtail/promtail.yml:/etc/promtail/promtail.yml:ro + - /var/run/docker.sock:/var/run/docker.sock:ro + - /var/lib/docker/containers:/var/lib/docker/containers:ro + - promtail_positions:/var/promtail + depends_on: + - loki + restart: unless-stopped + + tempo: + image: grafana/tempo:2.6.0 + container_name: spendhound_tempo + command: -config.file=/etc/tempo/tempo.yml + volumes: + - ./tempo/tempo.yml:/etc/tempo/tempo.yml:ro + - tempo_data:/var/tempo + restart: unless-stopped + flower: image: mher/flower:2.0 container_name: spendhound_flower @@ -220,3 +311,7 @@ volumes: postgres_data: prometheus_data: grafana_data: + alertmanager_data: + loki_data: + promtail_positions: + tempo_data: diff --git a/frontend/src/app/privacy/page.tsx b/frontend/src/app/privacy/page.tsx index 8178ceb..46b821c 100644 --- a/frontend/src/app/privacy/page.tsx +++ b/frontend/src/app/privacy/page.tsx @@ -126,6 +126,15 @@ export default function PrivacyPolicyPage() { delivery. Governed by Resend's Privacy Policy.

+
+

Sentry — Error monitoring

+

+ We use Sentry to capture application errors and exceptions. Error reports contain + stack traces, request URLs, and an anonymous user identifier (a random ID, never + your email or financial data). Data is processed on Sentry's EU infrastructure + (ingest.de.sentry.io). Governed by Sentry's Privacy Policy. +

+

LLM Providers — Receipt & statement parsing

diff --git a/grafana/loki-queries.md b/grafana/loki-queries.md new file mode 100644 index 0000000..7bcf3c0 --- /dev/null +++ b/grafana/loki-queries.md @@ -0,0 +1,137 @@ +# Loki / LogQL Reference Queries + +Run these in Grafana -> Explore -> Loki datasource. + +## Filter by service + +```logql +{service="backend"} +{service="celery_worker"} +{service="celery_beat"} +{service="frontend"} +``` + +## Errors and exceptions + +```logql +# Any error string across all services +{service=~".+"} |= "error" + +# Python exceptions (traceback) +{service="backend"} |= "Traceback" +{service="celery_worker"} |= "Traceback" + +# Structlog level=error (inline JSON parse - slower but accurate) +{service="backend"} | json | level="error" + +# Celery task failures +{service="celery_worker"} |= "FAILURE" +{service="celery_worker"} |= "exception" +``` + +## Receipt extraction + +```logql +# All receipt task activity +{service="celery_worker"} |= "receipt" + +# Extraction failures +{service="celery_worker"} |= "receipt" |= "error" + +# Confidence below threshold (sent to review) +{service="backend"} |= "needs_review" +``` + +## LLM / AI calls + +```logql +# All LLM provider calls +{service="backend"} |= "llm" + +# LLM errors or timeouts +{service="backend"} |= "llm" |= "error" +{service="backend"} |= "LLMTimeout" +``` + +## Auth and security + +```logql +# Failed logins / auth errors +{service="backend"} |= "auth" |= "error" + +# Rate limit hits +{service="backend"} |= "rate_limit" + +# New user signups +{service="backend"} |= "user.created" +``` + +## Monthly reports + +```logql +# All report task activity +{service="celery_worker"} |= "report_task" + +# Successful deliveries +{service="celery_worker"} |= "report_task.sent" + +# Failures and retries +{service="celery_worker"} |= "report_task.retry" +{service="celery_worker"} |= "report_task.all_retries_exhausted" +``` + +## Redis / cache + +```logql +# Cache misses and hits +{service="backend"} |= "cache.analytics" + +# Redis connection issues +{service="backend"} |= "cache.redis" +``` + +## Demo user + +```logql +# Demo reset activity +{service="celery_beat"} |= "demo" +{service="celery_worker"} |= "reset_demo" +``` + +## Inline JSON parsing + +structlog outputs JSON lines. Use `| json` to parse at query time and filter on any field: + +```logql +# Filter on any structlog key +{service="backend"} | json | level="warning" +{service="backend"} | json | level="error" + +# Extract and display a specific field +{service="backend"} | json | line_format "{{.event}} - {{.error}}" + +# Filter on nested event name +{service="celery_worker"} | json | event="report_task.sent" +``` + +## Volume / rate queries (for dashboard panels) + +```logql +# Log line rate per service over time +rate({service="backend"}[5m]) + +# Error rate +rate({service="backend"} |= "error" [5m]) + +# Count of receipt task completions in the last hour +count_over_time({service="celery_worker"} |= "extract_receipt" [1h]) +``` + +## Tips + +- `|=` is a case-sensitive substring filter (fast, uses index) +- `|~ "pattern"` is a regex filter (slower) +- `!=` excludes lines containing a string +- `| json` parses each line as JSON and promotes keys to labels - use for structured filtering +- `| line_format "{{.field}}"` reshapes the displayed log line +- In Grafana, set the time range in the top-right before running queries diff --git a/grafana/provisioning/datasources/loki.yaml b/grafana/provisioning/datasources/loki.yaml new file mode 100644 index 0000000..686f169 --- /dev/null +++ b/grafana/provisioning/datasources/loki.yaml @@ -0,0 +1,11 @@ +apiVersion: 1 + +datasources: + - name: Loki + type: loki + uid: loki + url: http://loki:3100 + access: proxy + isDefault: false + jsonData: + maxLines: 1000 diff --git a/grafana/provisioning/datasources/tempo.yaml b/grafana/provisioning/datasources/tempo.yaml new file mode 100644 index 0000000..603f230 --- /dev/null +++ b/grafana/provisioning/datasources/tempo.yaml @@ -0,0 +1,25 @@ +apiVersion: 1 + +datasources: + - name: Tempo + type: tempo + uid: tempo + url: http://tempo:3200 + access: proxy + isDefault: false + jsonData: + # "Logs" button on a trace view jumps to Loki with the same time range. + # filterByTraceID requires logs to emit a trace_id field; disabled for now. + tracesToLogsV2: + datasourceUid: loki + spanStartTimeShift: "-1m" + spanEndTimeShift: "1m" + filterByTraceID: false + filterBySpanID: false + # Service dependency graph panel uses Prometheus RED metrics. + serviceMap: + datasourceUid: prometheus + search: + hide: false + nodeGraph: + enabled: true diff --git a/loki/loki.yml b/loki/loki.yml new file mode 100644 index 0000000..012e49e --- /dev/null +++ b/loki/loki.yml @@ -0,0 +1,40 @@ +auth_enabled: false + +server: + http_listen_port: 3100 + grpc_listen_port: 9096 + +# Single-node deployment: all components run in one process. +common: + instance_addr: 127.0.0.1 + path_prefix: /loki + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + replication_factor: 1 + ring: + kvstore: + store: inmemory + +schema_config: + configs: + - from: 2024-01-01 + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: loki_index_ + period: 24h + +limits_config: + # 7-day retention. Compactor enforces this on a background schedule. + retention_period: 168h + +compactor: + working_directory: /loki/retention + retention_enabled: true + # Wait 2h after marking a chunk for deletion before actually removing it. + # Gives time to cancel an accidental config change. + retention_delete_delay: 2h + delete_request_store: filesystem diff --git a/pgbouncer/pgbouncer.ini b/pgbouncer/pgbouncer.ini new file mode 100644 index 0000000..214d275 --- /dev/null +++ b/pgbouncer/pgbouncer.ini @@ -0,0 +1,43 @@ +[databases] +; Map the database name clients connect to -> the actual Postgres host. +; "db" is the Postgres container name on the Docker Compose network. +spendhound = host=db port=5432 dbname=spendhound + +[pgbouncer] +listen_addr = * +listen_port = 5432 +; scram-sha-256 aligns with the Postgres 16 default and avoids md5 +; compatibility issues with newer asyncpg versions. +auth_type = scram-sha-256 +; Written at container startup by the entrypoint from $POSTGRES_USER / $POSTGRES_PASSWORD. +auth_file = /tmp/pgbouncer_userlist.txt + +; Session mode: each client connection keeps the same server connection for +; its entire lifetime. This gives full compatibility with asyncpg - no +; prepared-statement conflicts, no session-state resets between transactions. +; Transaction mode was causing asyncpg to see inconsistent server state +; (DateStyle, TimeZone, codec OID mappings) after DISCARD ALL ran between +; transactions on different server connections, producing intermittent errors. +pool_mode = session +; 30 server connections: enough for the SQLAlchemy pool (20) + Celery (1-2) +; + headroom for overflow connections and background tasks. +default_pool_size = 30 +min_pool_size = 2 +reserve_pool_size = 5 +max_client_conn = 200 + +; Clean up session state when a client disconnects so the server connection +; can be safely re-used by the next client. +server_reset_query = DISCARD ALL + +server_check_query = SELECT 1 +server_check_delay = 30 + +; asyncpg sends these in the connection startup handshake. PgBouncer rejects +; unknown startup parameters without this setting; application_name is also +; sent by asyncpg/SQLAlchemy in some configurations. +ignore_startup_parameters = extra_float_digits,options,application_name + +log_connections = 0 +log_disconnections = 0 +log_pooler_errors = 1 diff --git a/postgres/Dockerfile b/postgres/Dockerfile new file mode 100644 index 0000000..bb00bd4 --- /dev/null +++ b/postgres/Dockerfile @@ -0,0 +1,29 @@ +# Custom Postgres image for SpendHound production. +# Extends pgvector:pg16 with WAL-G (continuous backup to Cloudflare R2) +# and supercronic (daily base backup cron job in the backup sidecar). +# +# amd64 only - WAL-G binaries are architecture-specific. +# Update the WALG_VERSION build arg to upgrade WAL-G. + +FROM pgvector/pgvector:pg16 + +ARG WALG_VERSION=v3.0.3 +ARG SUPERCRONIC_VERSION=v0.2.33 + +RUN set -eux; \ + apt-get update; \ + apt-get install -y --no-install-recommends ca-certificates curl; \ + \ + # WAL-G PostgreSQL variant - download the raw binary directly (no tar needed) + curl -fsSL \ + "https://github.com/wal-g/wal-g/releases/download/${WALG_VERSION}/wal-g-pg-ubuntu-20.04-amd64" \ + -o /usr/local/bin/wal-g; \ + \ + # supercronic: cron runner for the backup sidecar container + curl -fsSL \ + "https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-amd64" \ + -o /usr/local/bin/supercronic; \ + \ + chmod +x /usr/local/bin/wal-g /usr/local/bin/supercronic; \ + apt-get purge -y --auto-remove curl; \ + rm -rf /var/lib/apt/lists/* diff --git a/postgres/backup-cron b/postgres/backup-cron new file mode 100644 index 0000000..df85693 --- /dev/null +++ b/postgres/backup-cron @@ -0,0 +1,13 @@ +# WAL-G daily base backup schedule for SpendHound. +# Runs inside the wal-g-backup sidecar container via supercronic. +# +# 02:00 UTC daily: +# 1. Take a full base backup and upload to Cloudflare R2. +# 2. Delete base backups beyond the 7 most recent (and associated WAL). +# +# WAL segments are archived continuously by Postgres archive_command - this +# cron only handles the periodic base backup that WAL segments are anchored to. +# On first deployment, trigger a manual backup before this schedule runs: +# docker exec spendhound_walg_backup_prod wal-g backup-push "$PGDATA" + +0 2 * * * wal-g backup-push "$PGDATA" && wal-g delete retain FULL 7 --confirm diff --git a/prometheus.prod.yml b/prometheus.prod.yml index d41bcaf..79685e9 100644 --- a/prometheus.prod.yml +++ b/prometheus.prod.yml @@ -2,6 +2,15 @@ global: scrape_interval: 15s evaluation_interval: 15s +alerting: + alertmanagers: + - static_configs: + - targets: + - alertmanager:9093 + +rule_files: + - /etc/prometheus/alert_rules.yml + scrape_configs: - job_name: spendhound_backend metrics_path: /metrics @@ -10,4 +19,4 @@ scrape_configs: - backend:8000 authorization: type: Bearer - credentials_file: /run/metrics_token + credentials_file: /tmp/metrics_token diff --git a/prometheus.yml b/prometheus.yml index 7dec21f..87d791a 100644 --- a/prometheus.yml +++ b/prometheus.yml @@ -2,6 +2,15 @@ global: scrape_interval: 15s evaluation_interval: 15s +alerting: + alertmanagers: + - static_configs: + - targets: + - alertmanager:9093 + +rule_files: + - /etc/prometheus/alert_rules.yml + scrape_configs: - job_name: spendhound_backend metrics_path: /metrics diff --git a/prometheus/alert_rules.yml b/prometheus/alert_rules.yml new file mode 100644 index 0000000..c50c31f --- /dev/null +++ b/prometheus/alert_rules.yml @@ -0,0 +1,58 @@ +groups: + - name: spendhound + rules: + # ── Critical ─────────────────────────────────────────────────────────────── + + - alert: BackendDown + # Prometheus built-in 'up' metric is 0 when the scrape target is unreachable. + expr: up{job="spendhound_backend"} == 0 + for: 2m + labels: + severity: critical + annotations: + summary: SpendHound backend is unreachable + description: >- + Prometheus cannot scrape the backend for more than 2 minutes. + The container may be down or the health check is failing. + + - alert: HighErrorRate + # Rate of 5xx responses as a fraction of all responses over 5 minutes. + # NaN (no traffic) is not > 0.05, so this does not fire when idle. + expr: | + ( + rate(http_requests_total{status=~"5.."}[5m]) + / + rate(http_requests_total[5m]) + ) > 0.05 + for: 2m + labels: + severity: critical + annotations: + summary: High HTTP 5xx error rate + description: >- + More than 5% of HTTP requests are returning 5xx errors + over the last 5 minutes. + + # ── Warning ──────────────────────────────────────────────────────────────── + + - alert: CeleryQueueBackedUp + expr: receipt_queue_depth > 5 + for: 5m + labels: + severity: warning + annotations: + summary: Celery receipt queue is backed up + description: >- + {{ $value | humanize }} tasks have been waiting in the Celery queue + for more than 5 minutes. The worker may be overloaded or crashed. + + - alert: LLMLatencyHigh + expr: histogram_quantile(0.95, rate(llm_response_seconds_bucket[5m])) > 30 + for: 5m + labels: + severity: warning + annotations: + summary: LLM p95 latency is high + description: >- + p95 LLM response time is {{ $value | humanizeDuration }}. + The AI provider may be degraded or the model is overloaded. diff --git a/promtail/promtail.yml b/promtail/promtail.yml new file mode 100644 index 0000000..e1bd74c --- /dev/null +++ b/promtail/promtail.yml @@ -0,0 +1,28 @@ +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +# Tracks read position in each log file so Promtail resumes after a restart +# without re-sending old logs. +positions: + filename: /var/promtail/positions.yaml + +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: + - job_name: docker + docker_sd_configs: + - host: unix:///var/run/docker.sock + refresh_interval: 5s + relabel_configs: + # Strip the leading slash from container names (/spendhound_backend_prod -> spendhound_backend_prod). + - source_labels: [__meta_docker_container_name] + regex: '/?(.*)' + target_label: container + # Expose the compose service name (backend, celery_worker, etc.) as a label. + - source_labels: [__meta_docker_container_label_com_docker_compose_service] + target_label: service + # stdout vs stderr. + - source_labels: [__meta_docker_container_log_stream] + target_label: stream diff --git a/tempo/tempo.yml b/tempo/tempo.yml new file mode 100644 index 0000000..9df4555 --- /dev/null +++ b/tempo/tempo.yml @@ -0,0 +1,32 @@ +# Grafana Tempo - single-binary mode, local filesystem storage. +# Accepts OTLP over gRPC (4317) and HTTP (4318). +# The app uses HTTP; gRPC is available for other senders (e.g. OTel Collector). + +server: + http_listen_port: 3200 + +distributor: + receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + +ingester: + # Flush blocks to storage after 5 min so traces appear in search quickly. + max_block_duration: 5m + +storage: + trace: + backend: local + local: + path: /var/tempo/blocks + wal: + path: /var/tempo/wal + +compactor: + compaction: + # Keep traces for 7 days - matches Loki log retention. + block_retention: 168h