Description
The alerter does not use the server's derived-metric facility. It
reimplements rate and delta logic in roughly 1200 lines of
hand-written SQL in
alerter/src/internal/database/metric_registry.go. Searching
alerter/src for the server's _per_sec machinery returns nothing.
The server's implementation lives in
server/src/internal/metrics/query.go. The two are free to drift, and
already have.
Why this matters now
The most significant divergence: the alerter's deltas are
per-sample, not per-second. Every delta threshold is therefore
implicitly scaled by the probe's collection_interval_seconds. An
operator retuning a probe interval silently rescales the alert
threshold, with no warning and no change to the rule.
checkpoint_warning is the clearest example — its unit is labelled
checkpoints, but the value is checkpoints per probe interval.
This also means the correctness work planned for the server's rate
derivation (per-dimension deltas before aggregation, the stats_reset
guard, gap handling) will not benefit the alerter unless it is
duplicated a second time.
Suggested direction
Either extract the rate derivation into a package both services
consume, or normalise the alerter's deltas to per-second so thresholds
become interval-independent. The former is more work but stops the
drift; the latter is a smaller change that fixes the immediate hazard.
At minimum, thresholds whose unit is per-probe-interval should say so
in the rule metadata, and changing a probe interval should warn about
affected rules.
Related smaller items
required_extension is stored on every rule but never enforced by the
alerter; the only references are the scan targets at
alerter/src/internal/database/queries.go:142 and :420. Rules gated
on system_stats, spock or pg_stat_statements are evaluated
unconditionally and rely on the metric query returning zero rows.
Benign today, but it makes "extension missing" indistinguishable from
"condition resolved", which interacts badly with the false-resolution
behaviour tracked separately.
table_bloat_ratio (collector/src/database/schema.go:2700) is
misnamed and duplicative. It computes n_dead_tup / n_live_tup * 100,
which is a dead-tuple ratio rather than bloat, and overlaps
dead_tuple_ratio (:2724) which uses n_dead / (n_live + n_dead).
Two rules, near-identical quantities, different denominators,
thresholds of 50 and 20, and different categories.
Description
The alerter does not use the server's derived-metric facility. It
reimplements rate and delta logic in roughly 1200 lines of
hand-written SQL in
alerter/src/internal/database/metric_registry.go. Searchingalerter/srcfor the server's_per_secmachinery returns nothing.The server's implementation lives in
server/src/internal/metrics/query.go. The two are free to drift, andalready have.
Why this matters now
The most significant divergence: the alerter's deltas are
per-sample, not per-second. Every delta threshold is therefore
implicitly scaled by the probe's
collection_interval_seconds. Anoperator retuning a probe interval silently rescales the alert
threshold, with no warning and no change to the rule.
checkpoint_warningis the clearest example — its unit is labelledcheckpoints, but the value is checkpoints per probe interval.This also means the correctness work planned for the server's rate
derivation (per-dimension deltas before aggregation, the
stats_resetguard, gap handling) will not benefit the alerter unless it is
duplicated a second time.
Suggested direction
Either extract the rate derivation into a package both services
consume, or normalise the alerter's deltas to per-second so thresholds
become interval-independent. The former is more work but stops the
drift; the latter is a smaller change that fixes the immediate hazard.
At minimum, thresholds whose unit is per-probe-interval should say so
in the rule metadata, and changing a probe interval should warn about
affected rules.
Related smaller items
required_extensionis stored on every rule but never enforced by thealerter; the only references are the scan targets at
alerter/src/internal/database/queries.go:142and:420. Rules gatedon
system_stats,spockorpg_stat_statementsare evaluatedunconditionally and rely on the metric query returning zero rows.
Benign today, but it makes "extension missing" indistinguishable from
"condition resolved", which interacts badly with the false-resolution
behaviour tracked separately.
table_bloat_ratio(collector/src/database/schema.go:2700) ismisnamed and duplicative. It computes
n_dead_tup / n_live_tup * 100,which is a dead-tuple ratio rather than bloat, and overlaps
dead_tuple_ratio(:2724) which usesn_dead / (n_live + n_dead).Two rules, near-identical quantities, different denominators,
thresholds of 50 and 20, and different categories.