feat: add deterministic synthetic OOM injection hooks - #7
Draft
yjyang62 wants to merge 3 commits into
Draft
Conversation
Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
【合入来源】
【修改方案】
HCCL_OOM_TRIGGER_MODE一次/持续触发,支持共享NPU_OOM_TRIGGER_COUNT作为自动计数触发 fallback,并保留触发文件作为可选调试手段。NPU_ALLOCATOR_OOM_TRIGGER_COUNT、共享NPU_OOM_TRIGGER_COUNT、一次/持续模式,按设备默认只触发一次。HCCL_OOM_COUNT、共享NPU_OOM_TRIGGER_COUNT,覆盖推理稳态无新增大分配、HCCL 计数不命中的路径。HCCL_OOM_COUNT=1、NPU_ALLOCATOR_OOM_TRIGGER_COUNT=0、HCCL_OOM_TRIGGER_COUNT=0,服务运行中无需 touch 文件或执行额外命令。OutOfMemoryError,错误信息包含NPU out of memory/Failed to allocate memory/Injected ... OOM,不实际占用 HBM,便于 MindIE 快恢恢复。【资料变更】
涉及。新增并更新
docs/oom_fault_injection.md,说明 PD 推理阶段推荐使用自动计数触发:在 MindIE 启动环境中配置 op 级 OOM 注入,P/D 节点进入推理后由 NPU op 计数自行触发,全 rank/全卡默认各触发一次。【接口变更】
涉及环境变量级调试/故障注入接口:
NPU_OOM_TRIGGER_COUNTNPU_OOM_TRIGGER_MODENPU_OOM_TRIGGER_FILE(仅作为可选调试手段)NPU_OP_OOM_TRIGGER_FILENPU_ALLOCATOR_OOM_TRIGGER_FILEHCCL_OOM_TRIGGER_FILEHCCL_OOM_COUNTNPU_ALLOCATOR_OOM_TRIGGER_COUNTNPU_OP_OOM_TRIGGER_MODENPU_ALLOCATOR_OOM_TRIGGER_MODEHCCL_OOM_TRIGGER_MODE【功能验证】
git diff --checkpython3 -m py_compile test/npu/test_fault_mode.pyInvalid device ID[\s\S]字符串的 SyntaxWarning,与本次新增代码无关。test_synthetic_allocator_oom_inject和test_synthetic_shared_oom_count_inject看护合成 OOM 自动计数注入入口。【CheckList】