diff --git a/.DS_Store b/.DS_Store new file mode 100644 index 0000000..cc464aa Binary files /dev/null and b/.DS_Store differ diff --git a/.gitignore b/.gitignore index dca7f4c..3336bd5 100644 --- a/.gitignore +++ b/.gitignore @@ -3,4 +3,5 @@ __pycache__ .mypy_cache scrape_configs.yml !vars/scrape_configs.yml +docker/volumes .venv diff --git a/@ b/@ new file mode 100644 index 0000000..652af04 --- /dev/null +++ b/@ @@ -0,0 +1 @@ +Please uninstall the existing Ansible and retry running this script. See installation instructions provided by Ansible: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html diff --git a/docker/.DS_Store b/docker/.DS_Store new file mode 100644 index 0000000..5a7d9ad Binary files /dev/null and b/docker/.DS_Store differ diff --git a/docker/README.md b/docker/README.md index c99b8e7..f78db74 100644 --- a/docker/README.md +++ b/docker/README.md @@ -8,8 +8,12 @@ metrics about the performance of selected nodes and send alerts if something is ## How does this work? The stack selects IC nodes and collects metrics from each node, saving them -in a local Prometheus database every few seconds. This database is -queryable through a Grafana deployed side-to-side with Prometheus. +in a local VictoriaMetrics database every few seconds. This database is +queryable through a Grafana instance deployed side-by-side with VictoriaMetrics. + +Additionally, a node rewards scheduler automatically collects daily node provider +rewards data from the IC Node Rewards Canister and stores it in VictoriaMetrics, +providing historical tracking and visualization of rewards metrics. Nodes of the Internet Computer all make available to the public a series of metrics (in [Prometheus text format](https://github.com/Prometheus/docs/blob/777846211d502a287ab2b304cb515dc779de3474/content/docs/instrumenting/exposition_formats.md#text-based-format)) @@ -67,9 +71,9 @@ Example command for node provider Dfinity Stiftung for data center se1: docker compose -f ./docker-compose.tools.yaml run --rm prom-config-builder tools/prom-config-builder/prom_config_builder.py --node-provider-id bvcsg-3od6r-jnydw-eysln-aql7w-td5zn-ay5m6-sibd2-jzojt-anwag-mqe --dc-id se1 ``` -Once that executes, you should be able to see a new file at `./config/prometheus/config.yaml`. +Once that executes, you should be able to see a new file at `./config/victoria-metrics/config.yaml`. This file contains the definitions for the scraping targets. It will be slightly different -for each node provider and each data center. It is not versione controlled and you can +for each node provider and each data center. It is not version controlled and you can always recreate it with running the above command if you lose it, or delete it. ### Contact points @@ -107,17 +111,33 @@ read *Troubleshooting* ## Usage Once started, you will see the following applications: -* Prometheus - http://localhost:9090 +* VictoriaMetrics - http://localhost:9090 * Grafana - http://localhost:3000 - default creds can be see in `./config/grafana/grafana.ini` * Service discovery - http://localhost:8000 -After 5-10 minutes you should see targets discovered in prometheus on the [targets page](http://localhost:9090/targets?search=). Initially, they might apear in red and if you keep monitoring they should -slowly start getting blue, which means that the targets are successfuly scraped. +After 5-10 minutes you should see targets discovered in VictoriaMetrics. The metrics are being +collected and stored successfully when you can query them through Grafana dashboards. You should also see some data incoming in the grafana [sample dashboard for the node exporter](http://localhost:3000/d/1/node-exporter?orgId=1&from=now-3h&to=now&timezone=utc&var-datasource=prometheus&var-instance=$__all&var-diskdevice=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B). Here, you can see various information about the performance of nodes and their health. +### Node Rewards Dashboard + +The stack includes a comprehensive Node Provider Rewards Dashboard that displays: +* **Provider Overview** - Total providers, nodes, and rewards calculations +* **Rewards Trends** - Historical tracking of base vs adjusted rewards +* **Node-Level Metrics** - Individual node failure rates and performance +* **Subnet Metrics** - Subnet-level failure rates and node distribution +* **Analysis & Insights** - Top providers, efficiency rankings, and reward penalties + +The node rewards scheduler automatically: +* Backfills the last 40 days of historical rewards data on first startup +* Pushes new rewards data daily at 00:05 UTC +* Downloads the latest `dre` binary from GitHub releases (v0.7.0) + +Access the dashboard at: http://localhost:3000/d/node-rewards + [Alerting tab](http://localhost:3000/alerting/list) will show `obs alert evaluations` which will contain a list of preconfigured alerts. Here you can see if any of the alerts are in in a problematic state. Most of the time they should be in `Normal` @@ -141,6 +161,11 @@ To access the stack remotely you can do the following: ssh -L 3000:localhost:3000 -L 9090:localhost:9090 -L 8000:localhost:8000 ``` +This will forward: +* Port 3000 - Grafana web interface +* Port 9090 - VictoriaMetrics API +* Port 8000 - Service discovery + Example command with all parameters: ```bash ssh -L 3000:localhost:3000 -L 9090:localhost:9090 -i ~/.ssh/priv_key.pem myuser@192.168.15.15 @@ -202,43 +227,56 @@ curl http://localhost:8000/prom/targets?node_provider_id=&dc_i *NOTE*: The initial sync of service discovery may take up to 15 minutes! Syncing will be clearly logged in the multiservice discovery. -### Prometheus +### VictoriaMetrics -#### No targets visible in targets view +VictoriaMetrics is the time-series database that stores all metrics from IC nodes +and node rewards data. -If you don't see anything in the [prometheus targets view](http://localhost:9090/targets?search=), -that means that prometheus failed to receive targets from the service discovery. +#### Checking VictoriaMetrics health -To check the logs run: +To verify VictoriaMetrics is running properly: +```bash +curl http://localhost:9090/-/ready +``` + +You should see: `VictoriaMetrics is Ready.` + +To check the logs: ```bash # From the same folder of this README -docker compose logs prometheus +docker compose logs victoriametrics ``` -Check if you can see your nodes by running the following command: +#### Querying metrics + +You can query metrics using the VictoriaMetrics API: ```bash -curl http://localhost:8000/prom/targets?node_provider_id=&dc_id= +# Check available metrics +curl 'http://localhost:9090/api/v1/label/__name__/values' + +# Query specific metric +curl 'http://localhost:9090/api/v1/query?query=latest_nodes_count' ``` -#### Targets visible but are being shown in read +### Node Rewards Scheduler -You should now see 4 jobs: -* `host_node_exporter` -* `node_exporter` -* `orchestrator` -* `replica` +The node rewards scheduler runs as a separate container and automatically collects +rewards data from the IC Node Rewards Canister. -If any of them are shown in read it means that some of the targets (or all of them) -are failing to be scraped. You can see that from the logs as well: +#### Checking scheduler status + +To view scheduler logs: ```bash # From the same folder of this README -docker compose logs prometheus +docker compose logs node-rewards-scheduler ``` -This means that the prometheus scraper cannot reach the nodes it is trying -to scrape. It can be because the workstation for this observability stack -isn't in the same network subnet as the nodes, or due to other network -issues. +On first startup, you should see: +* Binary download from GitHub releases +* 40-day backfill process (may take 10-20 minutes) +* Cron job setup for daily updates at 00:05 UTC + +``` ### Stack restart @@ -254,9 +292,14 @@ To make a full clean restart (or partial) you can do the following: * Stop the stack: `docker compose -f ./docker-compose.yaml down` * Clean the volumes. You don't have to clean everything, pick just ones that you wish to restart fully: - * prometheus: `rm -rf ./volumes/prometheus/` - * grafana: `rm -rf ./volumes/grafana/` - * multiservice discovery: `rm -rf ./volumes/msd/` + * VictoriaMetrics: `rm -rf ./volumes/victoriametrics/` + * Grafana: `rm -rf ./volumes/grafana/` + * Multiservice discovery: `rm -rf ./volumes/msd/` + * Node rewards scheduler: `rm -rf ./volumes/node-rewards-scheduler/` * Reset the folder structure: `git checkout -- ./volumes/` * Run the stack again: `docker compose -f ./docker-compose.yaml up -d` +**Note**: Cleaning VictoriaMetrics volumes will delete all historical metrics data, +including node rewards data. The node rewards scheduler will automatically backfill +the last 40 days when restarted. + diff --git a/docker/config/grafana/provisioning/alerting/notification_policy.yaml b/docker/config/grafana/provisioning/alerting/notification_policy.yaml index 450be44..75a37fa 100644 --- a/docker/config/grafana/provisioning/alerting/notification_policy.yaml +++ b/docker/config/grafana/provisioning/alerting/notification_policy.yaml @@ -1,11 +1,8 @@ apiVersion: 1 policies: - - receiver: Obs contact points - object_matchers: - - ['severity', '=~', 'critical'] + - receiver: grafana-default-email group_by: - alertname - ic_node - job - diff --git a/docker/config/grafana/provisioning/dashboards/node-rewards/node-rewards-dashboard.json b/docker/config/grafana/provisioning/dashboards/node-rewards/node-rewards-dashboard.json new file mode 100644 index 0000000..c876dd8 --- /dev/null +++ b/docker/config/grafana/provisioning/dashboards/node-rewards/node-rewards-dashboard.json @@ -0,0 +1,1983 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "victoriametrics1d" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "id": 6, + "links": [], + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 100, + "panels": [], + "title": "Provider Overview", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "dateTimeAsIso" + }, + "overrides": [ + { + "matcher": { + "id": "byType", + "options": "number" + }, + "properties": [ + { + "id": "unit", + "value": "dateTimeAsIso" + } + ] + } + ] + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 0, + "y": 1 + }, + "id": 101, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "/^Value$/", + "values": false + }, + "showPercentChange": false, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "last_over_time(governance_latest_reward_event_timestamp_seconds[35d]) * 1000", + "format": "table", + "instant": false, + "range": true, + "refId": "A" + } + ], + "title": "Last Rewards Distribution", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Total number of providers tracked", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 6, + "y": 1 + }, + "id": 102, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "max" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "count(latest_nodes_count offset -1d)", + "range": true, + "refId": "A" + } + ], + "title": "Total Providers", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Total nodes across all providers", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 12, + "y": 1 + }, + "id": 103, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "sum(last_over_time(latest_nodes_count[$__range] offset -1d))", + "refId": "A" + } + ], + "title": "Total Nodes", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Total adjusted rewards across all providers (XDR Permyriad)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "purple", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 18, + "y": 1 + }, + "id": 104, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum(sum_over_time(latest_total_adjusted_rewards_xdr_permyriad[$__range] offset -1d)) / 10000", + "range": true, + "refId": "A" + } + ], + "title": "Total Adjusted Rewards (XDR Permyriad)", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Comprehensive provider metrics overview", + "fieldConfig": { + "defaults": { + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "footer": { + "reducers": [] + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "provider_id" + }, + "properties": [ + { + "id": "custom.width", + "value": 180 + }, + { + "id": "links", + "value": [ + { + "title": "View Provider Details", + "url": "/d/${__dashboard.uid}?var-provider=${__value.text}&${__url_time_range}" + } + ] + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Base Total" + }, + "properties": [ + { + "id": "displayName", + "value": "Base Total (XDR)" + }, + { + "id": "unit", + "value": "short" + }, + { + "id": "custom.width", + "value": 140 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Adjusted Total" + }, + "properties": [ + { + "id": "displayName", + "value": "Adjusted Total (XDR)" + }, + { + "id": "unit", + "value": "short" + }, + { + "id": "custom.width", + "value": 160 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Difference" + }, + "properties": [ + { + "id": "displayName", + "value": "Adj - Base Diff" + }, + { + "id": "unit", + "value": "short" + }, + { + "id": "custom.cellOptions", + "value": { + "type": "color-text" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "green", + "value": 0 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Efficiency %" + }, + "properties": [ + { + "id": "displayName", + "value": "Rewards" + }, + { + "id": "unit", + "value": "percent" + }, + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "gauge" + } + }, + { + "id": "max", + "value": 100 + }, + { + "id": "min", + "value": 0 + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "orange", + "value": 90 + }, + { + "color": "yellow", + "value": 95 + }, + { + "color": "green", + "value": 98 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Nodes" + }, + "properties": [ + { + "id": "displayName", + "value": "Total Nodes" + }, + { + "id": "unit", + "value": "short" + }, + { + "id": "custom.width", + "value": 100 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 5 + }, + "id": 105, + "options": { + "cellHeight": "sm", + "enablePagination": false, + "showHeader": true, + "sortBy": [ + { + "desc": false, + "displayName": "Rewards" + } + ] + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "latest_nodes_count{provider_id=~\"$provider\"} offset -1d", + "format": "table", + "instant": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by (provider_id) (sum_over_time(latest_total_base_rewards_xdr_permyriad{provider_id=~\"$provider\"}[$__range])) / 10000", + "format": "table", + "instant": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by (provider_id) (sum_over_time(latest_total_adjusted_rewards_xdr_permyriad{provider_id=~\"$provider\"}[$__range])) / 10000", + "format": "table", + "instant": true, + "refId": "C" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "(sum by (provider_id) (sum_over_time(latest_total_adjusted_rewards_xdr_permyriad{provider_id=~\"$provider\"}[$__range])) - sum by (provider_id) (sum_over_time(latest_total_base_rewards_xdr_permyriad{provider_id=~\"$provider\"}[$__range]))) / 10000", + "format": "table", + "instant": true, + "refId": "D" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "(sum by (provider_id) (sum_over_time(latest_total_adjusted_rewards_xdr_permyriad{provider_id=~\"$provider\"}[$__range])) / sum by (provider_id) (sum_over_time(latest_total_base_rewards_xdr_permyriad{provider_id=~\"$provider\"}[$__range]))) * 100", + "format": "table", + "instant": true, + "refId": "E" + } + ], + "title": "Provider Rewards Summary (Latest)", + "transformations": [ + { + "id": "joinByField", + "options": { + "byField": "provider_id", + "mode": "outer" + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "Time 1": true, + "Time 2": true, + "Time 3": true, + "Time 4": true, + "Time 5": true, + "__name__": true, + "instance": true, + "job": true + }, + "indexByName": { + "Value #A": 1, + "Value #B": 2, + "Value #C": 3, + "Value #D": 4, + "Value #E": 5, + "provider_id": 0 + }, + "renameByName": { + "Value #A": "Nodes", + "Value #B": "Base Total", + "Value #C": "Adjusted Total", + "Value #D": "Difference", + "Value #E": "Efficiency %" + } + } + } + ], + "type": "table" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 15 + }, + "id": 200, + "panels": [], + "title": "Provider Rewards Trends", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Base vs Adjusted rewards over time for selected provider(s)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "XDR Permyriad", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*Base.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*Adjusted.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 16 + }, + "id": 201, + "options": { + "legend": { + "calcs": [ + "last", + "mean" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "latest_total_base_rewards_xdr_permyriad{provider_id=~\"$provider\"} offset -1d", + "legendFormat": "Base - {{provider_id}} ", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "latest_total_adjusted_rewards_xdr_permyriad{provider_id=~\"$provider\"} offset -1d", + "hide": false, + "instant": false, + "legendFormat": "Adjusted - {{provider_id}}", + "range": true, + "refId": "B" + } + ], + "title": "Base vs Adjusted Rewards Over Time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Rewards efficiency percentage (Adjusted/Base * 100)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Efficiency %", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "mappings": [], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 16 + }, + "id": 202, + "options": { + "legend": { + "calcs": [ + "last", + "mean", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "sortBy": "Last", + "sortDesc": false + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "(latest_total_adjusted_rewards_xdr_permyriad{provider_id=~\"$provider\"} offset -1d / latest_total_base_rewards_xdr_permyriad{provider_id=~\"$provider\"} offset -1d) * 100", + "legendFormat": "{{provider_id}} - Efficiency %", + "refId": "A" + } + ], + "title": "Provider Rewards Efficiency Over Time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Node counts over time", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Nodes", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 25 + }, + "id": 203, + "options": { + "legend": { + "calcs": [ + "last", + "mean" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "latest_nodes_count{provider_id=~\"$provider\"} offset -1d", + "legendFormat": "{{provider_id}} - Total Nodes", + "refId": "A" + } + ], + "title": "Node Counts Over Time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Rewards penalty/loss per provider (Base - Adjusted)", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "red", + "mode": "fixed" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": true, + "axisColorMode": "text", + "axisLabel": "Penalty (XDR Permyriad)", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "fieldMinMax": false, + "mappings": [], + "max": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "#F2495C", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 25 + }, + "id": 204, + "options": { + "legend": { + "calcs": [ + "last", + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "(latest_total_adjusted_rewards_xdr_permyriad{provider_id=~\"$provider\"} offset -1d- latest_total_base_rewards_xdr_permyriad{provider_id=~\"$provider\"} offset -1d)", + "legendFormat": "{{provider_id}} - Penalty", + "range": true, + "refId": "A" + } + ], + "title": "Provider Rewards Penalty (Base - Adjusted)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 34 + }, + "id": 300, + "panels": [], + "title": "Node-Level Metrics (Assigned)", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Original failure rate by node", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Failure Rate", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 5, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 35 + }, + "id": 301, + "options": { + "legend": { + "calcs": [ + "last", + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "label_replace(latest_original_failure_rate{provider_id=~\"$provider\",node_id=~\"$node\"} offset -1d, \"node_short\", \"$1\", \"node_id\", \"([^-]+)-([^-]+)-([^-]+).*\")", + "legendFormat": "{{node_short}}", + "range": true, + "refId": "A" + } + ], + "title": "Node Original Failure Rate", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Relative failure rate by node", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Relative Failure Rate", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 5, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 0.1 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 35 + }, + "id": 302, + "options": { + "legend": { + "calcs": [ + "last", + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "label_replace(latest_relative_failure_rate{provider_id=~\"$provider\",node_id=~\"$node\"} offset -1d, \"node_short\", \"$1\", \"node_id\", \"([^-]+)-([^-]+)-([^-]+).*\")", + "legendFormat": "{{node_short}}", + "range": true, + "refId": "A" + } + ], + "title": "Node Relative Failure Rate", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Heatmap showing distribution of original failure rates across all nodes", + "fieldConfig": { + "defaults": { + "custom": { + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "scaleDistribution": { + "type": "linear" + } + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 44 + }, + "id": 303, + "options": { + "calculate": false, + "cellGap": 2, + "cellValues": { + "unit": "short" + }, + "color": { + "exponent": 0.5, + "fill": "dark-orange", + "mode": "scheme", + "reverse": false, + "scale": "exponential", + "scheme": "RdYlGn", + "steps": 128 + }, + "exemplars": { + "color": "rgba(255,0,255,0.7)" + }, + "filterValues": { + "le": 1e-9 + }, + "legend": { + "show": true + }, + "rowsFrame": { + "layout": "auto" + }, + "tooltip": { + "mode": "single", + "showColorScale": false, + "yHistogram": false + }, + "yAxis": { + "axisPlacement": "left", + "reverse": false + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "label_replace(latest_original_failure_rate{provider_id=~\"$provider\", node_id=~\"$node\"}, \"node_short\", \"$1\", \"node_id\", \"([^-]+)-([^-]+)-([^-]+).*\")", + "format": "time_series", + "legendFormat": "{{node_short}}", + "range": true, + "refId": "A" + } + ], + "title": "Original Failure Rate Heatmap", + "type": "heatmap" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Heatmap showing distribution of relative failure rates across all nodes.\nrelative failure rate = original failure rate - subnet failure rate", + "fieldConfig": { + "defaults": { + "custom": { + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "scaleDistribution": { + "type": "linear" + } + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 44 + }, + "id": 304, + "options": { + "calculate": false, + "cellGap": 2, + "cellValues": { + "unit": "short" + }, + "color": { + "exponent": 0.5, + "fill": "dark-orange", + "mode": "scheme", + "reverse": false, + "scale": "exponential", + "scheme": "RdYlGn", + "steps": 128 + }, + "exemplars": { + "color": "rgba(255,0,255,0.7)" + }, + "filterValues": { + "le": 1e-9 + }, + "legend": { + "show": true + }, + "rowsFrame": { + "layout": "auto" + }, + "tooltip": { + "mode": "single", + "showColorScale": false, + "yHistogram": false + }, + "yAxis": { + "axisPlacement": "left", + "reverse": false + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "label_replace(latest_relative_failure_rate{provider_id=~\"$provider\", node_id=~\"$node\"}, \"node_short\", \"$1\", \"node_id\", \"([^-]+)-([^-]+)-([^-]+).*\")", + "format": "time_series", + "legendFormat": "{{node_short}}", + "range": true, + "refId": "A" + } + ], + "title": "Relative Failure Rate Heatmap", + "type": "heatmap" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Detailed node metrics table", + "fieldConfig": { + "defaults": { + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "footer": { + "reducers": [] + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "node_id" + }, + "properties": [ + { + "id": "custom.width", + "value": 200 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Original Failure Rate" + }, + "properties": [ + { + "id": "unit", + "value": "percentunit" + }, + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "gauge" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 0.05 + }, + { + "color": "red", + "value": 0.1 + } + ] + } + }, + { + "id": "max", + "value": 0.2 + }, + { + "id": "min", + "value": 0 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Relative Failure Rate" + }, + "properties": [ + { + "id": "unit", + "value": "percentunit" + }, + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "gauge" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "yellow", + "value": 0.05 + }, + { + "color": "red", + "value": 0.1 + } + ] + } + }, + { + "id": "max", + "value": 0.2 + }, + { + "id": "min", + "value": 0 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "subnet_id" + }, + "properties": [ + { + "id": "custom.width", + "value": 180 + }, + { + "id": "links", + "value": [ + { + "title": "View Subnet Details", + "url": "/d/${__dashboard.uid}?var-subnet=${__value.text}&${__url_time_range}" + } + ] + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 53 + }, + "id": 305, + "options": { + "cellHeight": "sm", + "enablePagination": true, + "showHeader": true, + "sortBy": [ + { + "desc": false, + "displayName": "Original Failure Rate" + } + ] + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "late{provider_id=~\"$provider\",node_id=~\"$node\"} offset -1d", + "format": "table", + "instant": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "latest_relative_failure_rate{provider_id=~\"$provider\",node_id=~\"$node\"} offset -1d", + "format": "table", + "instant": true, + "refId": "B" + } + ], + "title": "Node Performance Details", + "transformations": [ + { + "id": "joinByField", + "options": { + "byField": "node_id", + "mode": "outer" + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "Time 1": true, + "Time 2": true, + "__name__": true, + "instance": true, + "job": true + }, + "indexByName": { + "Value #A": 3, + "Value #B": 4, + "node_id": 1, + "provider_id": 0, + "subnet_id": 2 + }, + "renameByName": { + "Value #A": "Original Failure Rate", + "Value #B": "Relative Failure Rate" + } + } + } + ], + "type": "table" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 63 + }, + "id": 400, + "panels": [], + "title": "Subnet Metrics", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Subnet failure rates over time", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Failure Rate", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 64 + }, + "id": 401, + "options": { + "legend": { + "calcs": [ + "last", + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.2.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "subnet_failure_rate{subnet_id=~\"$subnet\"} offset -1d", + "legendFormat": "{{subnet_id}}", + "refId": "A" + } + ], + "title": "Subnet Failure Rates Over Time", + "type": "timeseries" + } + ], + "preload": false, + "refresh": "30s", + "schemaVersion": 42, + "tags": [ + "node-rewards", + "provider", + "rewards", + "performance" + ], + "templating": { + "list": [ + { + "current": { + "text": "VictoriaMetrics1d", + "value": "victoriametrics1d" + }, + "includeAll": false, + "label": "Data Source", + "name": "DS_PROMETHEUS", + "options": [], + "query": "prometheus", + "refresh": 1, + "regex": "", + "type": "datasource" + }, + { + "allValue": ".*", + "current": { + "text": "All", + "value": [ + "$__all" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values(latest_nodes_count, provider_id)", + "includeAll": true, + "label": "Provider", + "multi": true, + "name": "provider", + "options": [], + "query": { + "query": "label_values(latest_nodes_count, provider_id)", + "refId": "StandardVariableQuery" + }, + "refresh": 2, + "regex": "", + "sort": 1, + "type": "query" + }, + { + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values(latest_original_failure_rate{provider_id=~\"$provider\"}, node_id)", + "includeAll": true, + "label": "Node", + "multi": true, + "name": "node", + "options": [], + "query": { + "query": "label_values(latest_original_failure_rate{provider_id=~\"$provider\"}, node_id)", + "refId": "StandardVariableQuery" + }, + "refresh": 2, + "regex": "", + "sort": 1, + "type": "query" + }, + { + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values(subnet_failure_rate, subnet_id)", + "includeAll": true, + "label": "Subnet", + "multi": true, + "name": "subnet", + "options": [], + "query": { + "query": "label_values(subnet_failure_rate, subnet_id)", + "refId": "StandardVariableQuery" + }, + "refresh": 2, + "regex": "", + "sort": 1, + "type": "query" + }, + { + "allowCustomValue": false, + "current": { + "text": "1762947155000", + "value": "1762947155000" + }, + "definition": "query_result(last_over_time(governance_latest_reward_event_timestamp_seconds[35d]))", + "hide": 2, + "includeAll": false, + "name": "last_rewards_seconds", + "options": [], + "query": { + "qryType": 3, + "query": "query_result(last_over_time(governance_latest_reward_event_timestamp_seconds[35d]))", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "/\\d+\\s+(\\d+)/", + "type": "query" + }, + { + "baseFilters": [], + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "filters": [], + "name": "Filters", + "type": "adhoc" + } + ] + }, + "time": { + "from": "now-30d", + "to": "now" + }, + "timepicker": { + "refresh_intervals": [ + "10s", + "30s", + "1m", + "5m", + "15m", + "30m", + "1h", + "2h", + "1d" + ] + }, + "timezone": "utc", + "title": "Node Provider Rewards Dashboard - Comprehensive", + "uid": "node-rewards-comprehensive", + "version": 2 +} \ No newline at end of file diff --git a/docker/config/grafana/provisioning/dashboards/observability.yaml b/docker/config/grafana/provisioning/dashboards/observability.yaml index 6648df9..31b0e08 100644 --- a/docker/config/grafana/provisioning/dashboards/observability.yaml +++ b/docker/config/grafana/provisioning/dashboards/observability.yaml @@ -11,3 +11,14 @@ providers: options: path: /etc/grafana/provisioning/dashboards/samples foldersFromFilesStructure: false + + - name: 'Node Rewards' + folder: 'Node Rewards' + folderUid: 'node-rewards' + type: file + disableDeletion: false + updateIntervalSeconds: 60 + allowUiUpdates: true + options: + path: /etc/grafana/provisioning/dashboards/node-rewards + foldersFromFilesStructure: false \ No newline at end of file diff --git a/docker/config/grafana/provisioning/datasources/observability.yaml b/docker/config/grafana/provisioning/datasources/observability.yaml index 825b528..23e4af6 100644 --- a/docker/config/grafana/provisioning/datasources/observability.yaml +++ b/docker/config/grafana/provisioning/datasources/observability.yaml @@ -8,3 +8,12 @@ datasources: url: http://$PROM_HOST:9090 jsonData: httpMethod: POST + - name: VictoriaMetrics1d + uid: victoriametrics1d + type: prometheus + access: proxy + url: http://$PROM_HOST:9090 + isDefault: true + jsonData: + httpMethod: POST + timeInterval: "1d" diff --git a/docker/config/node-rewards-scheduler/entrypoint.sh b/docker/config/node-rewards-scheduler/entrypoint.sh new file mode 100755 index 0000000..2ae2ba4 --- /dev/null +++ b/docker/config/node-rewards-scheduler/entrypoint.sh @@ -0,0 +1,118 @@ +#!/bin/bash +set -e + +echo "=====================================" +echo "Node Rewards Scheduler for VictoriaMetrics" +echo "=====================================" + +# Install dependencies +echo "Installing dependencies..." +apt-get update -qq +apt-get install -y -qq curl cron ca-certificates > /dev/null 2>&1 + +# Set up directories +mkdir -p /scheduler +cd /scheduler + + +# # Download dre binary if not already present +# if [ ! -f "/scheduler/.dre_installed" ]; then +# echo "Downloading dre v${DRE_VERSION} binary..." + +# # Detect architecture +# ARCH=$(uname -m) +# if [ "$ARCH" = "x86_64" ]; then +# DRE_BINARY="dre-x86_64-unknown-linux" +# elif [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then +# # Note: v0.7.0 only has x86_64 Linux binary, no ARM64 Linux binary available +# echo "⚠️ ARM64 Linux binary not available, using x86_64 (may require emulation)" +# DRE_BINARY="dre-x86_64-unknown-linux" +# else +# echo "❌ Unsupported architecture: $ARCH" +# exit 1 +# fi + +# # Download from GitHub releases +# DRE_URL="https://github.com/dfinity/dre/releases/download/v${DRE_VERSION}/${DRE_BINARY}" +# echo "Downloading from: $DRE_URL" + +# curl -L -o /usr/local/bin/dre "$DRE_URL" +# chmod +x /usr/local/bin/dre + +# # Verify it works +# if /usr/local/bin/dre --version; then +# echo "✅ dre v${DRE_VERSION} installed successfully" +# touch /scheduler/.dre_installed +# else +# echo "❌ Failed to install dre" +# exit 1 +# fi +# else +# echo "✅ dre already installed" +# fi + +# Use locally built dre binary mounted from host +echo "Using locally built dre binary from /dre-binary/dre" + +if [ -f "/dre-binary/dre" ]; then + cp /dre-binary/dre /usr/local/bin/dre + chmod +x /usr/local/bin/dre + + # Verify it works + if /usr/local/bin/dre --version; then + echo "✅ Local dre binary installed successfully" + else + echo "❌ Failed to run local dre binary" + fi +else + echo "❌ Local dre binary not found at /dre-binary/dre" + echo "Please build the binary with: cd /Users/pietro.di.marco/dre && cargo build --release --bin dre" +fi + +# Wait for VictoriaMetrics to be ready +echo "Waiting for VictoriaMetrics to be ready..." +until curl -s "${VICTORIA_METRICS_URL}/-/ready" > /dev/null 2>&1; do + echo " Waiting for VictoriaMetrics at ${VICTORIA_METRICS_URL}..." + sleep 2 +done +echo "✅ VictoriaMetrics is ready" + +# Backfill last 40 days on initial startup +if [ ! -f /scheduler/.backfill_done ]; then + echo "" + echo "=====================================" + echo "Starting backfill of last 40 days..." + echo "=====================================" + + for i in $(seq 40 -1 1); do + DATE=$(date -u -d "$i days ago" +%Y-%m-%d 2>/dev/null || date -u -v-${i}d +%Y-%m-%d) + echo "[$(printf "%2d" $((41 - i)))/40] Backfilling data for $DATE..." + + if /usr/local/bin/dre node-rewards push-to-victoria --date "$DATE" --victoria-url "${VICTORIA_METRICS_URL}"; then + echo " ✅ Successfully pushed data for $DATE" + else + echo " ⚠️ No data available or error for $DATE" + fi + done + + touch /scheduler/.backfill_done + echo "" + echo "✅ Backfill complete!" + echo "=====================================" +fi + +# Set up cron job for daily push at 00:05 UTC (yesterday's data) +echo "" +echo "Setting up daily cron job (00:05 UTC)..." +mkdir -p /var/log +touch /var/log/node-rewards-cron.log +echo "05 00 * * * /usr/local/bin/dre node-rewards push-to-victoria --victoria-url ${VICTORIA_METRICS_URL} >> /var/log/node-rewards-cron.log 2>&1" | crontab - + +echo "✅ Cron job configured" +echo "" +echo "=====================================" +echo "Scheduler is running. Logs:" +echo "=====================================" + +# Start cron in foreground +cron && tail -f /var/log/node-rewards-cron.log 2>/dev/null || cron -f diff --git a/docker/config/prometheus/.gitignore b/docker/config/victoria-metrics/.gitignore similarity index 100% rename from docker/config/prometheus/.gitignore rename to docker/config/victoria-metrics/.gitignore diff --git a/docker/config/prometheus/config-template.yaml b/docker/config/victoria-metrics/config-template.yaml similarity index 98% rename from docker/config/prometheus/config-template.yaml rename to docker/config/victoria-metrics/config-template.yaml index 25da4d1..9651235 100644 --- a/docker/config/prometheus/config-template.yaml +++ b/docker/config/victoria-metrics/config-template.yaml @@ -28,7 +28,6 @@ anchors: http_configs: &http_configs http_sd_configs: - url: http://multiservice-discovery:8000/prom/targets - refresh_interval: 15s global: # Change this to control how frequently prometheus diff --git a/docker/docker-compose.tools.yaml b/docker/docker-compose.tools.yaml index 1a48db6..b5742a1 100644 --- a/docker/docker-compose.tools.yaml +++ b/docker/docker-compose.tools.yaml @@ -4,6 +4,6 @@ services: context: . dockerfile: ./tools/prom-config-builder/prom-config-builder.Dockerfile volumes: - - ./config/prometheus:/config/prometheus + - ./config/victoria-metrics:/config/victoria-metrics - ./tools/prom-config-builder/:/tools/prom-config-builder diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml index 910997c..425b8a2 100644 --- a/docker/docker-compose.yaml +++ b/docker/docker-compose.yaml @@ -12,17 +12,19 @@ services: volumes: - ./volumes/msd:/msd - prometheus: - image: quay.io/prometheus/prometheus:v2.51.1 + victoriametrics: + image: victoriametrics/victoria-metrics:v1.97.1 network_mode: host volumes: - - ./config/prometheus:/config - - ./volumes/prometheus/:/prometheus + - ./config/victoria-metrics:/config + - ./volumes/victoriametrics:/victoria-metrics-data command: - - --config.file - - /config/config.yaml - - --storage.tsdb.path=/prometheus - - --storage.tsdb.retention.time=6m + - --storageDataPath=/victoria-metrics-data + - --httpListenAddr=:9090 + - --retentionPeriod=1y + - --promscrape.config=/config/config.yaml + - --promscrape.configCheckInterval=30s + - --enableTCP6=true user: "${UID}:${GID}" deploy: resources: @@ -31,6 +33,21 @@ services: reservations: memory: 12G + node-rewards-scheduler: + image: ubuntu:24.04 + platform: linux/amd64 + network_mode: host + environment: + VICTORIA_METRICS_URL: http://localhost:9090 + DRE_VERSION: "0.7.0" + volumes: + - ./volumes/node-rewards-scheduler:/scheduler + - ./config/node-rewards-scheduler/entrypoint.sh:/entrypoint.sh:ro + entrypoint: /entrypoint.sh + user: "${UID}:${GID}" + depends_on: + - victoriametrics + grafana: image: grafana/grafana:12.2.1 network_mode: host @@ -40,3 +57,5 @@ services: - ./config/grafana:/etc/grafana - ./volumes/grafana:/var/lib/grafana user: "${UID}:${GID}" + depends_on: + - victoriametrics diff --git a/docker/tools/prom-config-builder/prom_config_builder.py b/docker/tools/prom-config-builder/prom_config_builder.py index 8108efa..cd188fc 100644 --- a/docker/tools/prom-config-builder/prom_config_builder.py +++ b/docker/tools/prom-config-builder/prom_config_builder.py @@ -7,21 +7,21 @@ def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( "prom-config-builder", - description="Used to expand the configuration to actual prometheus config", + description="Used to expand the configuration to actual victoria-metrics config", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) parser.add_argument( "--template-path", dest="template_path", - default="/config/prometheus/config-template.yaml", + default="/config/victoria-metrics/config-template.yaml", help="Path to the config template that should be expanded", ) parser.add_argument( "--output-path", dest="output_path", - default="/config/prometheus/config.yaml", + default="/config/victoria-metrics/config.yaml", help="Path to where the output should be written", ) diff --git a/docker/volumes/.DS_Store b/docker/volumes/.DS_Store new file mode 100644 index 0000000..a8f0fa7 Binary files /dev/null and b/docker/volumes/.DS_Store differ diff --git a/docker/volumes/grafana/.gitignore b/docker/volumes/grafana/.gitignore deleted file mode 100755 index e769a26..0000000 --- a/docker/volumes/grafana/.gitignore +++ /dev/null @@ -1,5 +0,0 @@ -# Ignore everything in this directory -* - -# But keep this .gitignore file -!.gitignore diff --git a/docker/volumes/influxdb/.gitignore b/docker/volumes/influxdb/.gitignore new file mode 100644 index 0000000..a5baada --- /dev/null +++ b/docker/volumes/influxdb/.gitignore @@ -0,0 +1,3 @@ +* +!.gitignore + diff --git a/docker/volumes/node-rewards-scheduler/.gitignore b/docker/volumes/node-rewards-scheduler/.gitignore new file mode 100644 index 0000000..a5baada --- /dev/null +++ b/docker/volumes/node-rewards-scheduler/.gitignore @@ -0,0 +1,3 @@ +* +!.gitignore +