Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #29 +/- ##
=======================================
Coverage 73.18% 73.18%
=======================================
Files 22 22
Lines 1965 1965
=======================================
Hits 1438 1438
Misses 527 527
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Member
Author
|
Some additional details Compression results
Adaptive query tiers (RSL panel, Auto mode)
|
Member
Author
|
I verified that things work as expected with a full rebuild of all containers. Can be merged now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Database & Grafana query performance optimizations
With 130M data points (364 CMLs × 2 sublinks, 26 days at 10s resolution) on an 8 GB / 4 CPU VM, loading raw data in the real-time dashboard was slow. The zoomed-out 1h aggregate view was already fast. This PR addresses the raw data path through three changes:
1. PostgreSQL memory tuning (
docker-compose.yml)The database container previously ran with PostgreSQL defaults (128 MB
shared_buffers). Parameters are now tuned for the VM size, keeping recently-used data chunks in RAM and steering the query planner toward index scans over sequential scans.2. TimescaleDB compression (
database/init.sql)Chunks older than 7 days are compressed automatically via a background policy, using
(cml_id, sublink_id)as the segment key so queries for a single CML decompress only ~1/728th of a chunk. The current week stays uncompressed for zero-overhead real-time ingestion. At ~21–51× compression ratio on existing data, the entire compressed history fits inshared_buffers. This scales well: as new streams are added the compressed footprint grows slowly while the hot uncompressed window stays bounded at one week.3. Adaptive query bucketing in the real-time Grafana dashboard
Replaced the binary 1h-aggregate / raw-10s switch with a three-tier system based on the selected time range. For the middle tier (≤ 3 days, Auto mode), a single CTE scan computes MIN/MAX/AVG via
time_bucket('$__interval', time), matching the panel's pixel density — so the min/max fill band and avg line are generated at no extra query cost. TheRawmode is now explicit-only, preventing accidental slow queries on wide time ranges. Minor: disabled point rendering on the RSL panel to suppress dots that appeared on the band boundary at low data density.