NPU Async Connector: tests and docs

[ShwStone]

⚠️ WIP — do not merge. This is a large PR; it will be split/narrowed before review.

Summary

Consolidates the NPU async CAM connector's test coverage and documentation on top of the v0.26 port (#203). It bundles three things:

1. v0.26 unit-test fixes — repairs 28 stale/broken NPU unit tests.
2. [E2E]: Add CAMAsync connector coverage with DeepSeek-V2-Lite #232 implementation — adds the async-cam-eager E2E case for CAMAsyncAFDConnector with configurable topology.
3. Async connector features & runtime gotchas — a consolidated summary of the connector's behavior and the operational pitfalls we hit on real NPU hardware.

1. v0.26 unit-test fixes

The v0.26 upgrade left 28 NPU unit tests failing on real hardware while CPU CI silently skipped them (pytest.importorskip). Every failure was test-side. After the fixes, tests/unit -m "not gpu and not vllm_runtime" is green (~450 passed).

• Circular import (collection blocker) — restored tests/conftest.py with a vllm_ascend.ops preimport to break the device_op → ops → experts_selector cycle; without it, collection fails with ImportError: cannot import name 'DeviceOperator'.
• Stale fixtures — use_v2_model_runner, afd_config/model_config, _ffn_loop_error, tensor-typed dp_metadata (was a plain list), read-only use_dcp → dcp_size, and the now-required initialize_attn_backend(kv_cache_config).
• Cross-file pollution — test_npu_mla_graph.py swaps a fake module into sys.modules; when the real module had never been imported, teardown had no original to restore and the fake leaked into later tests. Fixed by pre-importing the real module.
• Fake-torch leak — a string-typed fake torch leaked into a real .to(dtype) call (Invalid device string: 'int64'); replaced with _FakeIntVector.

Details: docs/npu/NPU_UNIT_TEST_FIX_2026-08-12.zh-CN.md.

2. #232 — async-cam-eager E2E

Adds async-cam-eager to the refactored tests/e2e/ framework:

• CAMAsyncAFDConnector with async=true, compute_gate_on_attention=true, eager execution (the async connector has no graph path).
• Attention/FFN topology is now explicit and configurable via AFD_E2E_ATTENTION_DEVICES / AFD_E2E_FFN_DEVICES instead of the fixed 2A1F split.
• validate_topology relaxed for configurable topology; build_launch_order extracted so the async connector launches Attention before FFN.
• CPU-safe unit coverage for command construction, connector flags, launch order, and topology validation.

The case is NPU-only and not yet run on hardware (see TODO).

3. Async connector features & runtime gotchas

Features

• Token-balanced async MoE ubatching — Attention and FFN are decoupled into an async transfer path; FFN runs as a single worker loop.
• compute_gate_on_attention — the gate is evaluated on the Attention side.
• Eager-only — no CUDA/NPU graph capture support.
• Attention-before-FFN launch order — required for the async handshake.
• Configurable Attention/FFN topology — no fixed 2A1F.

Runtime gotchas

• FFN idle self-crash (EI0006) — same root cause on single and dual node; bisect confirmed the plugin code is innocent. Mitigation: send-ready immediately + HCCL_CONNECT_TIMEOUT=3600 + GPU_MEM_UTIL=0.8.
• Precision-capture boundaries — capturing tensors in the async-transfer path crashes token-split. Safe boundaries: layer_input / attn_output / final_hidden.
• token-split divergence — the global SFA RoPE buffer was overwritten by the stage-1 builder; fixed in 68bb799.
• Metadata dump crash — dump only tensor values; reading attn_metadata during dump crashes the FFN.
• Residual workers — VLLMWorker_DP linger after kill vllm and hold HBM; clean with npu-smi info + kill -9.
• Multi-machine — cross-node HCCL requires the same super-node; take the mask from ifconfig.
• GPU_MEM_UTIL 0.8 vs 0.9 — 0.9 leaves too little room for the HCCL_BUFFSIZE=4096 communication pool (~8.6 GB); 0.8 is safe.
• Debug switch — AFD_CAM_OP_IO_LOG=1 + ASCEND_LAUNCH_BLOCKING=1 to trace send/recv.
• Single-node reduced model — DP1TP8+EP8; CAM vendor env must be set.

TODO (before undrafting)

• Run async-cam-eager on real Ascend NPU and attach accuracy/topology/version evidence.
• Re-run tests/unit/test_e2e_runner.py and the fixed NPU unit tests in an environment with pytest.
• Decide CI selection for the NPU-only async-cam-eager case (pytest node ID vs @pytest.mark.npu).

---

🤖 Generated with Claude Code