Feat/swa suite - #57
Merged
Merged
Conversation
Feature A — Energy / power collection: - Add sample_power_watts() to each vendor platform plugin (nvidia, amd, ascend, moorethreads; google and apple return None for now) - Add get_active_plugin() shared helper in runners/platforms/__init__.py - Create loadgen/power.py: PowerSampler with background daemon thread - Wire PowerSampler into offline, online, burst, and sustained scenario timed windows in loadgen/loadgen.py - Populate existing power_watts_avg / power_watts_peak schema fields - Compute tokens_per_joule and energy_joules derived metrics in validate_submission.py and surface them in leaderboard/generate.py - Add unit tests in loadgen/tests/test_power.py Feature B — Arithmetic-intensity profiling: - Add tools/profile_intensity.py: standalone roofline profiler using torch FlopCounterMode and CUDA events for prefill/decode phases - Add results/profiling/README.md with paper cycle mapping No runner.py files modified; all runner hashes remain unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
PowerSampler is instantiated per-run, and each construction previously triggered get_active_plugin() which spawned subprocesses for every vendor SMI tool (nvidia-smi, rocm-smi, npu-smi, mthreads-gmi) in sequence. On an NVIDIA host this meant hundreds of redundant failing subprocesses across a full suite run — not correctness-affecting (all outside the timed window) but measurably slowing down wall-clock. Now both get_active_plugin() and _get_sample_fn() cache their result at module level, so the heavy vendor detection runs exactly once per process. Co-Authored-By: Claude <noreply@anthropic.com>
1. Refactor collect_env.py _collect_accelerators() to delegate to the shared get_active_plugin() — eliminates the duplicate vendor-detection loop. Single source of truth for platform resolution is now in runners/platforms/__init__.py. 2. Support CUDA_VISIBLE_DEVICES in GPU UUID format in nvidia.py's sample_power_watts(). Cross-references nvidia-smi UUID→index mapping and warns when UUIDs cannot be resolved (shared-node over-count). 3. Profiler env_ref now accepts --env-file <path> and stores the SHA-256 of the env_info.json content for provenance tracking, falling back to chip name + note when the flag is not provided. Co-Authored-By: Claude <noreply@anthropic.com>
…erMode recovery - loadgen: wrap interactive scenario with PowerSampler for energy parity across all scenarios - profile_intensity: add A800-SXM4-80GB peak specs + FlopCounterMode graceful fallback on GQA assertion errors - leaderboard: cache-bust leaderboard.js with content hash Co-Authored-By: Claude <noreply@anthropic.com>
Supports idempotent/resumable runs across SGLang → vLLM → profiling phases. Detects outdated results (missing power data) for re-run. Includes card configs for P1.1–P1.8 from the experiment TODO. Co-Authored-By: Claude <noreply@anthropic.com>
- Filter existing results by detected chip slug to avoid cross-chip confusion - Detect stale results (missing power_watts_avg) and auto-enable --force - Add conda environment support for SGLang runner Co-Authored-By: Claude <noreply@anthropic.com>
…0.1) Companion to Suite A: isolates sliding window attention effect on roofline intensity spectrum. Mistral-7B shares identical structural parameters with Llama-3-8B (32L/32H/8KVH/4096d) — only the attention mechanism differs. Design rationale: Mistral-7B chosen over newer SWA models (Gemma 3/4) because its standard mistral architecture ensures cross-platform compatibility (V100, T4, Ascend, TPU, AMD) — a core requirement of AccelMark's multi-vendor evaluation. Co-Authored-By: Claude <noreply@anthropic.com>
…, F) - New runner at runners/nvidia_tensorrt_llm_21323970/ implementing BenchmarkRunner ABC via TRT-LLM 0.21.0 PyTorch backend - trtllm_env.py: pre-import MPI/CUDA workarounds separate from runner.py to avoid hash changes from environment tuning - Supports BF16 only; quantization (FP8/W8A8/W8A16/W4A16) unsupported in TRT-LLM 0.21 PyTorch backend - Suite D (28K context) fails: torch.export graph capture hangs - Results on A800 vs vLLM: suite A -13%, suite C -11%, suite F -63% Demonstrates Type II effects (same HW, different SW stacks) - Adds TRT-LLM phases to batch_driver.py Co-Authored-By: Claude <noreply@anthropic.com>
Suite H uses Mistral-7B-Instruct-v0.1 (sliding_window=4096) to isolate SWA architecture effects vs Suite A (Llama-3-8B, full attention). vLLM 0.7.3: 2067/2069/2067 tok/s (conc=8/32/128) TRT-LLM 0.21: 2316/2314/2312 tok/s (conc=8/32/128) Key finding: TRT-LLM is 12% FASTER than vLLM on Mistral-7B, reversing the Suite A finding where vLLM was 13% faster on Llama-3-8B. Framework ranking depends on model architecture — a Type II effect. Also fixed Mistral config: added head_dim=128 (vLLM 0.7.3 requires explicit head_dim in config.json for Mistral models). Co-Authored-By: Claude <noreply@anthropic.com>
SGLang 0.4.10 on Mistral-7B-Instruct-v0.1 (A800): conc=8: 3434 tok/s conc=32: 3556 tok/s conc=128: 3557 tok/s SGLang is 72% faster than vLLM and 54% faster than TRT-LLM on SWA. All three frameworks rank differently on Mistral-7B (SGLang > TRT-LLM > vLLM), confirming that framework choice is a significant Type II effect. Also fixed: - SGLang runner: event loop init for uvloop compatibility - SGLang env: patched hf3fs client to skip missing .cpp JIT compile Co-Authored-By: Claude <noreply@anthropic.com>
P0-1: Replace profiler-based bytes with analytical DRAM calculation. Weight reads + KV cache (128 KiB/token) - closed-form from model geometry. Fixes Suite A decode bytes from 8.3 GB (physically impossible - less than weight alone) to 17.2 GB. P0-2: Apply causal-mask correction to SDPA FLOP counting. FlopCounterMode counts full SxS pairs; causal self-attention only computes S(S+1)/2. Fixes Suite D prefill FLOPs from 8.60e14 (+32% overcount) to 6.45e14. P0-3: Add blended arithmetic intensity and phase time fractions. Output now includes a per-request blended AI weighted by the suite's real dataset in/out length distribution. Suite A blended AI=53.3 -> bandwidth-bound (consistent with paper Table 3). P0-4: Fix ridge-point chip specs and add I* table utility. RTX 5090 corrected from 104.8 (FP32) to 838 TFLOPS (FP16 tensor). New --list-i-star CLI flag. 18/42 chips marked as verified. P1-1: Add runtime provenance note to profiling output documenting the HF-vs-vLLM measurement gap and why analytical bytes mitigate it. P2: Mark 18 benchmarked chips as verified in chip_specs.py; remaining 24 entries tagged as datasheet-only in I* table. Co-Authored-By: Claude <noreply@anthropic.com>
The --model flag may receive a local filesystem path for loading, but the output JSON should use the portable HuggingFace model ID from suite.json so results are shareable across machines. Co-Authored-By: Claude <noreply@anthropic.com>
…+ A100 intensity results - profiling: add FlashAttention-based activation traffic to analytical byte model (O(B·S·(4d+2·d_ff)) per layer), reduce prefill I from inflated values (Suite A: 7841→3138, Suite D: 32560→6299 FLOP/byte) - profiling: pass phase=prefill|decode to measure_dram_bytes() across all backends (nvidia, amd, apple, ascend) instead of seq_len≤1 heuristic - profiling: update runtime_note to document three-component byte model - Suite H: switch dataset to sharegpt_swa_v1 (10K tokens > sliding_window=4096) - Suite H: update description to document SWA KV cache framework support status - TRT-LLM: improve MPI prefix detection — try sys.prefix first, fallback gracefully; add single-node MPI config (shared memory, disable CUDA-aware) - Add A100-80GB intensity profiling results for suites A, C, D, F, H - .gitignore: narrow workspace exclusion to nips26_rebuttal/; add model.cache Co-Authored-By: Claude <noreply@anthropic.com>
…ty results Co-Authored-By: Claude <noreply@anthropic.com>
…_8d1b4efe) This SGLang result has no registered runner (implementation_id=None, runner_name=None) and would not display correctly on the leaderboard. Will be replaced with a properly registered SGLang Suite H submission. Co-Authored-By: Claude <noreply@anthropic.com>
- runners/meta.schema.json: add 'H' to suite_support properties - TRT-LLM: fix invalid status strings (complete→validated, untested→pending) - vLLM 0.7.3: add H=validated (committed Suite H results) - SGLang 0.4.10: add H=pending (results pending) Co-Authored-By: Claude <noreply@anthropic.com>
…-LLM) TRT-LLM 0.21 PyTorch backend does not support accuracy evaluation in this setup — the generation interface differs from vLLM/SGLang. Setting accuracy=null is the honest representation. Co-Authored-By: Claude <noreply@anthropic.com>
✅ AccelMark Validation: All submissions validSee the workflow run for details. |
validate_submission.py recomputes derived metrics (tokens_per_sec_per_watt, tokens_per_joule, energy_joules) when energy data is present. Co-Authored-By: Claude <noreply@anthropic.com>
- runners/nvidia_sglang_c43a8309/runner.py: revert asyncio event-loop fix. Editing runner.py breaks the content-hash folder name (c43a8309), orphaning 20+ pre-existing results that reference it. The fix was only needed for the exploratory SGLang Suite H run whose result was removed; it can be re-added in a dedicated PR that renames the runner folder. - tools/generate_platforms_matrix.py: add Suite H to SUITE_KEYS - README.md: regenerate platforms matrix (now includes H column) Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…, H) Ran the accuracy scenario (100-question subset) on A800 with TRT-LLM 0.21.0 PyTorch backend, matching the original run environment (torch 2.7.1+cu126, CUDA 12.6, driver 565.57.01). - Suite A (Llama-3-8B): score 0.61, delta +0.01 vs baseline - Suite C (Llama-3.1-8B, BF16): score 0.57, delta +0.01 - Suite F (Qwen2.5-0.5B): score 0.39, delta +0.01 - Suite H (Mistral-7B): score 0.43, baseline pending (delta null) All valid=True. Suite C stores accuracy per-format under bf16/accuracy/. Co-Authored-By: Claude <noreply@anthropic.com>
results/community/ is for benchmark submissions; results/profiling/ is the documented output dir for tools/profile_intensity.py (has its own README). - Remove stale A800 suite A/D intensity from community/ (newer versions already exist in results/profiling/ after the activation-byte model fix) - Move A100-40GB suite A/D intensity into results/profiling/ (no duplicate) Co-Authored-By: Claude <noreply@anthropic.com>
Two independent TRT-LLM runs on A800 both scored 0.43/100 on the accuracy subset, so this is a stable result, not noise. Set the bf16_baseline_score to 0.43 per the 'establish on first successful BF16 run' rule. Note the subset protocol scores systematically below full MMLU (cf. Qwen2.5-0.5B 0.39 vs 0.38 baseline), so 0.43 is the correct subset-level baseline even though full MMLU is ~0.55-0.60. Suite H accuracy delta is now 0.0 (score == baseline). Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type of change
Testing
# Commands used to verifyChecklist
result.jsonfiles (or I have explained the migration path)BenchmarkRunner, produces validresult.json, includes a reference resultvalidate_submission.pyupdated and all existing results still validateleaderboard/generate.pyproduces correct output on existing resultsRelated issues