Skip to content

[release-1.14] Fix hami vGPU scheduling failure in large and medium-scale clusters#5434

Open
Copilot wants to merge 2 commits into
release-1.14from
copilot/release-114
Open

[release-1.14] Fix hami vGPU scheduling failure in large and medium-scale clusters#5434
Copilot wants to merge 2 commits into
release-1.14from
copilot/release-114

Conversation

Copilot AI commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Cherry-pick of aec04d7 from master to release-1.14. The handshake annotation mechanism (volcano.sh/node-vgpu-handshake) caused vGPU scheduling failures in medium/large clusters due to timing inconsistencies between the scheduler and hami-dp.

Changes

  • pkg/scheduler/api/devices/config/vgpu.go: Remove VolcanoVGPUHandshake constant
  • pkg/scheduler/api/devices/nvidia/vgpu/device_info.go: Replace handshake annotation check in NewGPUDevices with direct node.Status.Allocatable checks for vgpu-number, vgpu-cores, and vgpu-memory
  • pkg/scheduler/api/devices/nvidia/vgpu/utils.go: Remove patchNodeAnnotations helper (no longer referenced)

Before → After

// Before: gated on a handshake annotation that could get stuck
handshake, ok := node.Annotations[deviceconfig.VolcanoVGPUHandshake]
if !ok {
    return nil
}
// ... timing-based state machine using "Requesting_" / "Deleted_" strings

// After: check allocatable resources directly
gpuNumberRes, gpuNumberExists := node.Status.Allocatable[v1.ResourceName(deviceconfig.VolcanoVGPUNumber)]
if !gpuNumberExists || gpuNumberRes.Value() == 0 {
    return nil
}
// ... similar checks for vgpu-cores and vgpu-memory

@volcano-sh-bot volcano-sh-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 11, 2026
@volcano-sh-bot volcano-sh-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jun 11, 2026
Signed-off-by: fanhy36 <fanhy36@chinaunicom.cn>
@volcano-sh-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jessestutler. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@volcano-sh-bot volcano-sh-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 11, 2026
Copilot AI changed the title [WIP] Remove handshake between scheduler and hami-dp cherry-pick: remove handshake between scheduler and hami-dp (release-1.14) Jun 11, 2026
Copilot AI requested a review from JesseStutler June 11, 2026 12:33
@JesseStutler JesseStutler changed the title cherry-pick: remove handshake between scheduler and hami-dp (release-1.14) [release-1.14] Fix hami vGPU scheduling failure in large and medium-scale clusters Jun 11, 2026
@JesseStutler JesseStutler marked this pull request as ready for review June 11, 2026 12:40
@volcano-sh-bot volcano-sh-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants