Skip to content

Import pta_oom_2.9.1 HCCL OOM fault injection from gitcode - #4

Draft
yjyang62 wants to merge 3 commits into
cursor/hccl-oom-fault-2bd5from
cursor/pta-oom-2.9.1-fbca
Draft

Import pta_oom_2.9.1 HCCL OOM fault injection from gitcode#4
yjyang62 wants to merge 3 commits into
cursor/hccl-oom-fault-2bd5from
cursor/pta-oom-2.9.1-fbca

Conversation

@yjyang62

Copy link
Copy Markdown
Owner

Summary

This branch copies the pta_oom_2.9.1 branch from https://gitcode.com/yangjinyang/pytorch into this repository.

Changes

  • ProcessGroupHCCL.cpp: Add configurable HCCL OOM fault injection via maybeThrowHcclOom() after collective/P2P HCCL calls. Trigger count is controlled by the HCCL_OOM_TRIGGER_COUNT environment variable (default: 6000).
  • NPUCachingAllocator.cpp: Remove the fixed-counter PTA malloc OOM injection (g_malloc_call_count at 60004).
  • HCCLUtils.hpp: Keep the original HCCL_CHECK_ERROR macro without macro-level OOM injection.

Usage

export HCCL_OOM_TRIGGER_COUNT=6000   # inject OOM after the 6001st HCCL op
# export HCCL_OOM_TRIGGER_COUNT=0     # disable injection

Source

  • Remote: https://gitcode.com/yangjinyang/pytorch
  • Branch: pta_oom_2.9.1
  • Tip commit: d8450a496 (hccloom)

Test plan

  • Set HCCL_OOM_TRIGGER_COUNT and verify HCCL OOM is injected at the expected collective count
  • Verify fast recovery (stop_devicerestart_devicereinit_process_group) works after injected OOM
  • Confirm HCCL_OOM_TRIGGER_COUNT=0 disables injection
Open in Web Open in Cursor 

yjyang62 added 3 commits June 12, 2026 16:56
Signed-off-by: yangjinyang <yangjinyang5@huawei.com>
Signed-off-by: yangjinyang <yangjinyang5@huawei.com>
Signed-off-by: yangjinyang <yangjinyang5@huawei.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant