fix(metrics): classify per-GPU series as gauges not counters

[shingonoide]

Summary

Per-GPU system metrics (gpuN, gpuN_mem) are emitted as monotonic OpenTelemetry counters instead of point-in-time gauges. Downstream, the reported "GPU Utilization %" accumulates the running sum of utilization samples, so it grows without bound and exceeds 100% (a near-straight ramp from 0% to 155% over ~5 minutes was observed on a single-GPU instance).

Root cause

OpenTelemetryClient decides gauge-vs-counter by membership in a static _GAUGE_METRICS set. The per-GPU keys are emitted with dynamic names (gpu0, gpu1, gpu0_mem, ...), so they are never in that set and fall through to create_counter, recorded via instrument.add(value) — accumulating the sum of samples over time. cpu/mem/fps are created as gauges and are unaffected.

Fix

Add name-based classification for the dynamic per-GPU series (gpuN / gpuN_mem) alongside the existing static set, so they are created as gauges. Genuine counters (frame_count, megapx_count, *_ts, ...) and the health gauges (gpu_usage_percent, gpu_accessible) are unchanged.

The fix also corrects the data type of the gpuN_mem series. No downstream read-path change is required — the read path faithfully displays whatever the emitter sends.

Tests

Added TestGpuGaugeClassification in tests/test_timing_metrics.py covering gpu0 / gpu10 / gpu0_mem / gpu12_mem (gauge) and non-matches like foo_gpu0, frame_count. Full module: 39 passed.

[leandrobmarinho]

Review

Summary: Classifies the dynamic per-GPU series (gpuN, gpuN_mem) as gauges instead of counters, fixing the GPU Utilization % that accumulated a running sum and climbed past 100%.

Looks good. No blocking issues found. The root cause is right — dynamic per-GPU keys never appear in the static _GAUGE_METRICS set, so they fell through to create_counter + add() and accumulated — and the _is_gauge_metric classifier is precise:

• core.startswith("gpu") and core[3:].isdigit() (after stripping an optional _mem) matches gpu0 / gpu10 / gpu0_mem / gpu12_mem (incl. multi-GPU two-digit indices) but correctly rejects foo_gpu0 (uses startswith, not substring), bare gpu ("".isdigit() is False), gpux, and gpu0_foo. Health gauges gpu_usage_percent / gpu_accessible don't match the digit pattern, so their classification is untouched, and genuine counters (frame_count, *_ts) are unaffected.

Tests are thorough — they pin the classifier boundaries, assert gpu0/gpu0_mem create observable gauges (no counters), that repeated updates store the latest value rather than add()-ing a sum, and that frame_count still records via add(). This is also the correct emitter-side complement to the read-side VRAM/memory classification in plainsight-api#944.