Recently, our Prometheus instances have been experiencing severe stability issues, including continuous disk space exhaustion and frequent crashes.
Upon investigation, we traced the root cause to a high-cardinality explosion originating from OpenWebUI’s OTEL /metrics endpoint.
Modern frontend assets in OpenWebUI (JS/CSS chunks) are bundled with unique cache-busting hashes (e.g., /assets/index-C3b7x9a2.js). The instrumentation inside OpenWebUI captures the raw, literal path using request.url.path rather than generic route templates. As a result, Prometheus generates a completely unique time-series line for every single static asset chunk requested by users, quickly exhausting the disk allocation. Prometheus storage PVC fills up unexpectedly due to the somewhat unbounded nature of the problem. With every update of OpenWebUI we deploy a whole set of unique time-series gets created and in a matter of a few updates (1-2) our PVC is starved.
Since we deploy OpenWebUI as an upstream dependency, we can fix this either upstream, by fixing the OpenWebUI application code or downstream, by filtering it out via our monitoring configuration. A third option is to drop metrics from openwebui wholesale if no value is gained from keeping them around.
To reproduce this issue run the following PromQL to see the footprint: count_values("path", http_server_duration_milliseconds_bucket)
Proposed todo:
- Do we need the metrics at all?
- Implement dropping of (specific static asset) metrics in our Prometheus configuration.
- Submit a PR to OpenWebUI that fixes the problem at the source by labeling the static asset metrics as one group.
Recently, our Prometheus instances have been experiencing severe stability issues, including continuous disk space exhaustion and frequent crashes.
Upon investigation, we traced the root cause to a high-cardinality explosion originating from OpenWebUI’s OTEL /metrics endpoint.
Modern frontend assets in OpenWebUI (JS/CSS chunks) are bundled with unique cache-busting hashes (e.g., /assets/index-C3b7x9a2.js). The instrumentation inside OpenWebUI captures the raw, literal path using request.url.path rather than generic route templates. As a result, Prometheus generates a completely unique time-series line for every single static asset chunk requested by users, quickly exhausting the disk allocation. Prometheus storage PVC fills up unexpectedly due to the somewhat unbounded nature of the problem. With every update of OpenWebUI we deploy a whole set of unique time-series gets created and in a matter of a few updates (1-2) our PVC is starved.
Since we deploy OpenWebUI as an upstream dependency, we can fix this either upstream, by fixing the OpenWebUI application code or downstream, by filtering it out via our monitoring configuration. A third option is to drop metrics from openwebui wholesale if no value is gained from keeping them around.
To reproduce this issue run the following PromQL to see the footprint:
count_values("path", http_server_duration_milliseconds_bucket)Proposed todo: