deployment/docker: Add alerting rules#1
Merged
Merged
Conversation
hagen1778
reviewed
Jun 26, 2026
| annotations: | ||
| # todo: add link to dashboard | ||
| summary: "High cardinality for job {{ $labels.by_job }}" | ||
| description: "Job {{ $labels.by_job }} has {{ $value }} active series over the last {{ $labels.interval }}, exceeding the 5000 series threshold." |
There was a problem hiding this comment.
ActiveSeries is VM-related term, other DBs may not have it. Just say "cardinality" or "number of unique series"
Comment on lines
+17
to
+23
| - alert: 'Cardinality10xWithin1h' | ||
| expr: | | ||
| ( | ||
| max without(job, instance) (cardinality_estimate{group_by_keys!="__global__",group_by_keys!="__group__"}) | ||
| / | ||
| max without(job, instance) (cardinality_estimate{group_by_keys!="__global__",group_by_keys!="__group__"} offset 1h) | ||
| ) > 10 |
There was a problem hiding this comment.
I'd avoid x10 and x24 like alerts, as they're very specific to environment and selected time interval.
What I would like to know whether cardinality grows over time. That can be measured with deriv but you have to account for multiple things:
- Daily or seasonal patterns - cardinality can grow and shring during the day. The daily growth should be accounted in a formula
- A burst-like spike in cardinality growth
So I'd suggest something like:
(deriv(cardinality_estimate[1h]) > burst_threshold
and
deriv(cardinality_estimate[5m]) > burst_threshold)
or
(deriv(cardinality_estimate[6h]) > low_threshold
and
deriv(cardinality_estimate[30m]) > low_threshold)
Additionally you might want to look at https://prometheus.io/blog/2015/06/18/practical-anomaly-detection/
The general idea behind it is to detect cardinality growth on short and long ranges.
Member
Author
|
Merged by accident, reopned in #2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.