Skip to content

fix: stop exporting stale metrics for completed or failed pods#131

Open
maishivamhoo123 wants to merge 1 commit into
Project-HAMi:mainfrom
maishivamhoo123:fix/stale-pod-metrics
Open

fix: stop exporting stale metrics for completed or failed pods#131
maishivamhoo123 wants to merge 1 commit into
Project-HAMi:mainfrom
maishivamhoo123:fix/stale-pod-metrics

Conversation

@maishivamhoo123

Copy link
Copy Markdown
Member

Enhance pod validation to check both pod phase and container running status.

Previously, volcano-vgpu-monitor would export metrics for completed or failed
pods as long as the Pod object existed in Kubernetes. This caused stale metrics
to remain visible in Prometheus.

Changes:

  • Verify pod is in PodRunning phase before considering it valid
  • Check that target container is actively running (not terminated or waiting)
  • Parse cache directory names to safely extract pod UID and container name
  • Add detailed logging for validation results
  • Preserve existing 300-second grace period for cache cleanup

Fixes #130

Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
@hami-robot hami-robot Bot requested a review from archlitchi May 25, 2026 09:20
@hami-robot

hami-robot Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: maishivamhoo123
Once this PR has been reviewed and has the lgtm label, please assign archlitchi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hami-robot hami-robot Bot requested a review from SataQiu May 25, 2026 09:20
@hami-robot hami-robot Bot added the size/L label May 25, 2026
return false
}

for _, containerStatus := range pod.Status.ContainerStatuses {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shouldn't validate the container status, because if container crashes, the folder will be deleted, that will make any attempt to restart the container failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

volcano-vgpu-monitor exports stale pod metrics for completed pods

2 participants