Skip to content

Add automatic full-card PTA OOM injection (no PTA recovery) - #5

Draft
yjyang62 wants to merge 11 commits into
cursor/pta-oom-2.9.1-fbcafrom
cursor/guaranteed-pta-oom-c058
Draft

Add automatic full-card PTA OOM injection (no PTA recovery)#5
yjyang62 wants to merge 11 commits into
cursor/pta-oom-2.9.1-fbcafrom
cursor/guaranteed-pta-oom-c058

Conversation

@yjyang62

@yjyang62 yjyang62 commented Jun 24, 2026 •
edited by cursor Bot
Loading

Copy link
Copy Markdown
Owner

Summary

Simplified PTA OOM injection to timer-only. All count-based logic removed.

Configuration

Env Meaning
PTA_OOM_TRIGGER_AFTER_MINUTES=6 Throw OOM 6 minutes after PTA load
PTA_OOM_TRIGGER_AFTER_SECONDS=360 Same, explicit seconds
PTA_OOM_TIMER=1 Enable with default 360s (6 min)
PTA_OOM_DEBUG=1 Log timer progress to ascend log

No env set = injection disabled (no behavior change).

Behavior

  • Timer starts when PTA .so loads
  • Background thread marks expiry at deadline
  • Next hook (malloc, sync, op, memcpy, etc.) throws full-card OutOfMemoryError
  • P and D use the same mechanism — just set the timer env on whichever instance you want to test

Removed

  • PTA_OOM_TRIGGER_COUNT
  • PTA_OOM_FORWARD_SYNC_TRIGGER_COUNT
  • PTA_OOM_SKIP_WARMUP_COUNT
  • PTA_OOM_INJECT
  • recordPtaOomProgress() / maybeThrowPtaOomOnForwardBoundary() APIs
Open in Web Open in Cursor 

@cursor cursor Bot changed the title Add guaranteed full-card PTA OOM injection (PTA_OOM_INJECT) Add guaranteed full-card OOM injection (PTA + graph replay) Jun 24, 2026
@cursor cursor Bot changed the title Add guaranteed full-card OOM injection (PTA + graph replay) Add guaranteed OOM injection (Python + PTA + graph replay) Jun 24, 2026
@cursoragent @yjyang62
Add Python-level inject_oom for inference OOM testing
64b229b 
Provide a single top-level OutOfMemoryError injection point for workloads
such as Qwen 235B that bypass C++ allocator and HCCL hooks. Controlled via
NPU_INJECT_OOM_STEP and a call at the top of each inference iteration.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursor cursor Bot changed the title Add guaranteed OOM injection (Python + PTA + graph replay) Add Python-level inject_oom for inference OOM testing Jun 24, 2026
@cursor
cursor Bot force-pushed the cursor/guaranteed-pta-oom-c058 branch from 795edcf to 64b229b Compare June 24, 2026 04:00
@cursoragent @yjyang62
Add automatic PTA OOM injection at upper C++ layers
f70d250 
Remove Python inject_oom API. Introduce maybeThrowPtaOom() controlled by
PTA_OOM_TRIGGER_COUNT (default 6000), called from MakeSureQueueEmpty (async
queue drain, where OOM propagates upward) and malloc() entry. Skips graph
capture. Raises torch.OutOfMemoryError with PTA memory fault code.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursor cursor Bot changed the title Add Python-level inject_oom for inference OOM testing Add automatic PTA OOM injection at upper C++ layers Jun 24, 2026
@cursoragent @yjyang62
Make PTA OOM injection full-card fault
477da6f 
On trigger, mark all allocator blocks on the device unsafe and enable
unsafe-data checking before throwing OutOfMemoryError. Reset inject state
on restart_device and clear unsafe check flag after recovery.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursor cursor Bot changed the title Add automatic PTA OOM injection at upper C++ layers Add automatic full-card PTA OOM injection at upper C++ layers Jun 24, 2026
@cursoragent @yjyang62
Remove PTA OOM inject recovery hooks for MindIE fast recovery
94d2eb9 
Drop resetPtaOomInjectState, restart_device reset call, and unsafe-flag
clearing. Keep full-card OOM fault injection only; recovery is handled
externally by MindIE.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursor cursor Bot changed the title Add automatic full-card PTA OOM injection at upper C++ layers Add automatic full-card PTA OOM injection (no PTA recovery) Jun 24, 2026
@cursoragent @yjyang62
Raise default PTA_OOM_TRIGGER_COUNT to 400000 to skip warmup
4ab4cee 
Avoid injecting full-card OOM during model warmup; aligns with prior
malloc counter tuning for large models like Qwen 235B.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursor
cursor Bot force-pushed the cursor/guaranteed-pta-oom-c058 branch from adc18d3 to 4ab4cee Compare June 24, 2026 08:52
cursoragent and others added 6 commits June 24, 2026 08:54
@cursoragent @yjyang62
Add PTA OOM counter on P static graph execution paths
f01ec55 
Record progress in InnerRunOpApi, ExecFunc, ExecFuncOpApi, and
AclmdlRIExecuteAsync so prefill static-graph inference increments the
counter. Defer throw to malloc/MakeSureQueueEmpty via pending flag.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursoragent @yjyang62
Count PTA OOM at stream/device sync for Qwen P GE prefill
d3cf460 
TorchAir GE static graph bypasses OpCommand during prefill body; hook
AclrtSynchronizeStreamWithTimeout and npuSynchronizeDevice so each
prefill forward completion increments and can throw OOM on MindIE path.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursoragent @yjyang62
Guarantee PTA OOM on P prefill via forward boundary hooks
dfdccaa 
Add PTA_OOM_INJECT=1 for immediate full-card OOM on first forward
boundary (stream/device/event sync, queue drain, memcpy).

Add PTA_OOM_FORWARD_SYNC_TRIGGER_COUNT (default 10) separate from the
400k op counter so Qwen GE prefill can trigger without per-op PTA paths.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursoragent @yjyang62
Fix P prefill OOM: forward boundary mode when main counter disabled
f0aba97 
When PTA_OOM_TRIGGER_COUNT=0 (P instance config), route malloc/queue/op
hooks to the forward-boundary counter instead of no-op.

Add PTA_OOM_SKIP_WARMUP_COUNT, PTA_OOM_DEBUG, route stream sync through
hooked AclrtSynchronizeStreamWithTimeout, and hook D2H memcpy plus
allocator event sync as additional P prefill boundaries.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursoragent @yjyang62
Add timer-based PTA OOM trigger after configurable delay
94abaac 
Support PTA_OOM_TRIGGER_AFTER_SECONDS, PTA_OOM_TRIGGER_AFTER_MINUTES,
and PTA_OOM_TIMER=1 (default 6 minutes). Timer starts at PTA load and
throws full-card OOM on the next hook after the deadline.

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
@cursoragent @yjyang62
Remove count-based PTA OOM injection, keep timer only
d6b0ffa 
Delete PTA_OOM_TRIGGER_COUNT, forward sync counter, warmup skip, and
PTA_OOM_INJECT. All hooks now call maybeThrowPtaOom which only checks
the timer (PTA_OOM_TRIGGER_AFTER_SECONDS/MINUTES or PTA_OOM_TIMER=1).

Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@yjyang62 @cursoragent