Celery+redis+prom+grafana#24
Closed
sumdher wants to merge 3 commits into
Closed
Conversation
OpenTelemetry + Grafana Tempo: auto-instrument FastAPI, SQLAlchemy, HTTPX, Redis, and Celery; manual llm.complete spans in MeteredLLMProvider; separate service names (spendhound-api / spendhound-worker) visible in Tempo service map. WAL-G continuous Postgres backup to Cloudflare R2: archive_command streams WAL segments on every write, archive_timeout=60 caps max data loss to 60s, daily base backup sidecar (supercronic) at 02:00 UTC, 7-day retention, PITR-capable. PgBouncer connection pooler in session mode (scram-sha-256) between app and Postgres. Session mode required for asyncpg compatibility - transaction mode causes DISCARD ALL to reset session-level settings intermittently. Alertmanager with Slack webhook: BackendDown (critical) and HighErrorRate (warning) rules; inhibit_rule suppresses HighErrorRate when BackendDown fires. Loki + Promtail: structured log aggregation from Docker socket; trace IDs in logs become clickable links to Tempo traces. GDPR privacy page (Article 17 data erasure, data residency disclosure). Fix: Prometheus and Alertmanager write credential files to /tmp/ not /run/. Both images (prom/prometheus, prom/alertmanager) run as nobody; /run/ is root-owned and not writable. /tmp/ is world-writable (mode 1777). Fix: WAL-G backup sidecar uses absolute path /usr/local/bin/supercronic. Docker OCI exec resolves bare command names differently from shell which.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.