fix(metrics): emit gpu_usage_percent as float for Cloud Monitoring (FILTER-585)#113
Conversation
…ILTER-585) NVML reports GPU utilization as an integer, so gpu_usage_percent was emitted as INT64 whenever a GPU was active. The Cloud Monitoring descriptor for this gauge is type-locked to DOUBLE, so every active-GPU point was rejected with 'value type for metric must be DOUBLE, but is INT64' and silently dropped. Cast the value to float so it matches the descriptor. The per-device <filter>_gpuN counters and the INT64 gpu_accessible/camera_connected metrics are deliberately left untouched. Adds a regression test that drives the GPU poll path with an integer NVML reading and asserts a float result. Signed-off-by: Rui Andrada <randrada@plainsight.ai>
Signed-off-by: Rui Andrada <randrada@plainsight.ai>
…TER-585) Signed-off-by: Rui Andrada <randrada@plainsight.ai>
leandrobmarinho
left a comment
There was a problem hiding this comment.
Review
Summary: Casts gpu_usage_percent to float in Metrics.gpu_thread_func so an active GPU's integer gpu_util (gpu.py:71, gpu_util: int) no longer gets rejected by the DOUBLE-locked openfilter_gpu_usage_percent Cloud Monitoring descriptor (value type ... must be DOUBLE, but is INT64).
Verified the fix is correct and minimal: gpu['gpu0'] derives from the int gpu_util when a GPU is present, so the old code emitted an int; the cast forces DOUBLE. Scope is right — the per-device gpuN series and the legitimately INT64 descriptors (gpu_accessible, camera_connected) are untouched, and the downstream observability/client.py handles the value generically. No contract break, no reference/mutation concern.
FYI (non-blocking): test_gpu_usage_percent_is_float_when_gpu0_absent doesn't actually guard the regression — the no-gpu0 default 0.0 was already a float on both old and new code, so it passes either way; the gpu0-present test is the one that exercises the fix. No change needed to merge.
…-585) Signed-off-by: Rui Andrada <randrada@plainsight.ai>
lucasmundim
left a comment
There was a problem hiding this comment.
Reviewed: the float() cast is correct and narrowly scoped to the gpu_usage_percent summary field that maps to Cloud Monitoring's DOUBLE descriptor (openfilter_gpu_usage_percent). Per-device gpuN series and the legitimately-INT64 flags are correctly left untouched. VERSION bump, RELEASE.md format, and the new test all check out. LGTM.
📋 What does this PR do?
Casts the
gpu_usage_percentsystem-health metric tofloatso it is always emitted as a floating-point value.🔍 Why is this needed?
NVML reports GPU utilization as an integer, so
gpu_usage_percentwas emitted as an integer whenever a GPU was active (the float default0.0only applies when no GPU is present). The Google Cloud Monitoring descriptor for this gauge is type-locked toDOUBLE, so every active-GPU point was rejected by the backend with:and silently dropped (a partial, single-point rejection per export batch). The result is silent gaps in GPU-utilization data. Forcing the value to
floatmakes the emitted points match the existingDOUBLEdescriptor, restoring ingestion with no descriptor migration.The change is intentionally narrow: the per-device
<filter>_gpuNcounters and the legitimately-integergpu_accessible/camera_connectedmetrics are left untouched, since their descriptors are integer-typed and casting them would cause the opposite rejection.🧪 How was it tested?
tests/test_gpu_usage_percent_float.pydrives the GPU poll path with a mocked integer NVML reading and assertsgpu_usage_percentis afloatequal to55.0. It fails on the previous (un-cast) code and passes with this change.tests/test_gpu.py,tests/test_telemetry.py,tests/test_timing_metrics.py).🔗 Related Issues
FILTER-585
✅ Checklist
git commit -s)Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.