feat(observability): pgx + Asynq queue + DLQ metrics + Sentry/GlitchTip error tracking - #488
Merged
Merged
Conversation
Closes #165. Adds packages/go/db/metrics.go: a prometheus.Collector implementation that pulls pgxpool.Stat() at scrape time and emits the gonext_db_pool_* gauges + cumulative counters defined in docs/10-observability.md §5.3. Histograms for query duration (gonext_db_query_duration_seconds) and transaction duration (gonext_db_tx_duration_seconds) are push-based, fed via ObserveQuery / ObserveTx so callers wire them at the data-access layer. Replication lag (gonext_db_replication_lag_seconds) is gated on a caller-supplied ReplicaProber; deployments without a replica leave it nil and the gauge is skipped. ReplicaProberFunc adapts a plain function for callers that want a closure over a replica pool. Wires the collector into apps/api/cmd/server/main.go alongside the existing metrics registry so /metrics exposes the new series without any operator action. Signed-off-by: Tayeb Mokni <tayeb.mokni@gmail.com>
…rics Closes #172. The existing metrics.go covered processed/failed/inflight/unknown, which are sourced from handler-side middleware and only reflect the in-process worker. The cluster-wide queue state — pending tasks, processing lag, retry pool, archived (dead-letter) tasks, paused-ness — lives in Redis and is accessed through Asynq's Inspector API. Adds InspectorCollector: a prometheus.Collector that calls Inspector.GetQueueInfo for each configured queue on every scrape and emits gonext_jobs_queue_depth, gonext_jobs_queue_active, gonext_jobs_queue_lag_seconds, gonext_jobs_retries, gonext_jobs_dlq_size, and gonext_jobs_queue_paused. Failures are logged + counted on gonext_jobs_inspector_failures_total so operators can alert on a flapping Redis connection. Defines QueueInspector as the minimal interface the collector needs, so tests use a fakeInspector without standing up Redis. *asynq.Inspector satisfies the interface directly. Wires the collector into apps/worker/cmd/worker/main.go alongside the existing metrics registry, with the inspector handle registered with the shutdown orchestrator so it drains cleanly on SIGTERM. Signed-off-by: Tayeb Mokni <tayeb.mokni@gmail.com>
…e grouping Closes #202. Adds packages/go/observability/errortracker: a thin wrapper around getsentry/sentry-go that exposes Init / Capture / CaptureMessage / Recover / WithPluginSlug and isolates the rest of the codebase from sentry-go primitives (Hub, Scope, Event). Swapping to a different reporter later is a one-package change. Behavior is gated on the configured DSN (GONEXT_SENTRY_DSN env var, or Options.DSN). When unset the package installs a no-op tracker with zero overhead on the hot path; Shutdown is a no-op. When set, Init wires sentry-go with attach-stacktrace, AttachStacktrace=true, the build-info release, the config environment, and a hostname-derived ServerName. The returned Shutdown closer flushes the in-flight event queue with a 5s budget. Plugin-aware grouping: WithPluginSlug stamps a gonext.plugin.slug tag onto the event scope. Per-request middleware threads the slug on every plugin dispatch (in follow-up wiring); end-of-chain Capture calls inherit it via context, so events from plugin handlers group separately from host code in the Sentry dashboard. Wires the tracker into apps/api + apps/worker main.go with the Shutdown closer registered against the orchestrator — drains BEFORE the DB pool / Redis so events captured during shutdown still reach the ingestion endpoint over a live network. Setup failures are non-fatal (log a warning and continue without reporting), matching the existing tracing.Setup posture. Signed-off-by: Tayeb Mokni <tayeb.mokni@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #165, #172, #202.