attention backend sweep

.gitattributes

@@ -0,0 +1,3 @@
+*.sh   text eol=lf
+*.py   text eol=lf
+*.yaml text eol=lf

.gitignore

@@ -0,0 +1,167 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# results directory
+results/
+# test output files
+*.txt
+*.html
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

README.md

@@ -10,6 +10,7 @@ Each recipe is one `(model, hardware, set of tasks)` combination. The Buildkite
 workloads/        one YAML per (model, hardware) recipe
 lib/              orchestrator (run.sh), helpers, GPU profiles
 .buildkite/       pipeline bootstrap and step generator
+gen_report.py     generate HTML benchmark reports from results/
 CLAUDE.md         agent conventions and detailed Buildkite workflow
 ```
 
@@ -26,7 +27,7 @@ CLAUDE.md         agent conventions and detailed Buildkite workflow
 
 A recipe has top-level metadata plus up to three eval blocks:
 
-- **`vllm:`** — *how the server runs.* Defines what model to serve and how (`model`, `serve_args`, optional image/env overrides). Required.
+- **`vllm:`** — *how the server runs.* Defines what model to serve and how (`model`, `serve_args`, optional image/env overrides, optional `attention_backends` or `moe_backends` list). Required.
 - **`lm_eval:`** — *what accuracy to measure.* Lists lm-evaluation-harness tasks to run against the live server (e.g. `gsm8k`, `aime25`). Each task's score is saved under `results/<name>/<task-name>/`. Optional.
 - **`vllm_bench:`** — *what perf to measure.* Lists `vllm bench serve` configs (input/output lengths, concurrency, dataset). Raw JSON is saved and ingested into the perf dashboard. Optional.
 - **`bfcl:`** — *function-calling eval.* Runs [BFCL](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) test categories against the live server. Some models need `--enable-auto-tool-choice` and `--tool-call-parser` in `serve_args`. Results are transformed to lm_eval format and ingested as `bfcl_<category>` tasks. Optional.
@@ -47,6 +48,13 @@ vllm:                    # how the server is brought up
   serve_args: >-                        # appended to `vllm serve <model>`; word-split
     -dp 8 --enable-expert-parallel
     --trust-remote-code
+  attention_backends:                   # optional; list of VLLM_ATTENTION_BACKEND values
+    - FLASH_ATTN                        # when set, the full eval suite runs once per
+    - FLASHINFER                        # backend; results land in attn-<BACKEND>/ subdirs
+  # moe_backends:                       # optional; list of --moe-backend values
+  #   - default                         # "default" means no --moe-backend flag (vLLM picks)
+  #   - AITER                           # results land in moe-<BACKEND>/ subdirs
+  # attention_backends and moe_backends are mutually exclusive in a single workload
 
 lm_eval:                 # accuracy tasks (optional)
   model_args:            # workload-level defaults, merged into every task
@@ -85,6 +93,8 @@ vllm_bench:              # perf runs (optional) — fed to the perf dashboard
 
 A few things worth knowing:
 
+- **`vllm.attention_backends`** is an optional list of vLLM attention backend names (`FLASH_ATTN`, `FLASHINFER`, `XFORMERS`, `TRITON_ATTN`, `TRITON_MLA`, `ROCM_FLASH`, `PAGED_ATTENTION`,`ROCM_AITER_FA`,`ROCM_AITER_UNIFIED_ATTN`, `ROCM_ATTN`,`ROCM_AITER_MLA`,`ROCM_AITER_MLA_SPARSE`, `ROCM_AITER_TRITON_MLA`). When set, the orchestrator starts the server once per backend — adding `--attention-backend $ATTN_BACKEND` — and runs the complete eval suite (bench, lm_eval, bfcl) for each. Results are stored under `results/<name>/attn-<BACKEND>/` so every backend gets its own isolated output directory. Without this field, the server starts once with whatever attention backend vLLM selects by default and results go to `results/<name>/` as usual. See `workloads/attn-sweep-gpt-oss-120b-mi355x.yaml` for an example.
+- **`vllm.moe_backends`** is an optional list of vLLM `--moe-backend` values (`default`, `AITER`, `TRITON`, `FUSED_MOE`, `deep_gemm_mega_moe`). When set, the orchestrator starts the server once per backend — adding `--moe-backend $MOE_BACKEND` for non-default values — and runs the complete eval suite for each. Results are stored under `results/<name>/moe-<BACKEND>/`. The special value `"default"` means no `--moe-backend` flag (vLLM picks the backend automatically). **Do not include `--moe-backend` in `serve_args` when using this field** — the sweep script owns that flag. `attention_backends` and `moe_backends` are mutually exclusive in a single workload. See `workloads/moe_sweep_deepseek_r1_0528_mi355x.yaml` for an example.
 - **`gpu`** must match a key in `lib/gpu_profiles.yaml`. The profile sets the Buildkite queue, default image, HF cache path, and baseline env vars.
 - **`nightly`** controls only the nightly schedule. Recipes with `nightly: false` (or omitted) are still triggerable explicitly via the `WORKLOADS` env var.
 - **`lm_eval.tasks` is a list** because each entry runs as a separate `lm_eval` invocation — `--num_fewshot` is a single global flag, so different shot counts need separate runs. Each task's results land in `results/<name>/<task-name>/`.
@@ -138,6 +148,22 @@ A real run needs a GPU host with Docker, vLLM, and lm-eval available:
 
 Locally, you can smoke-test recipe changes without a GPU — see `CLAUDE.md` for the parser stub and shell-syntax checks.
 
+## Benchmark reports
+
+After a run completes, generate interactive HTML reports from the `results/` directory:
+
+```bash
+python3 gen_report.py
+```
+
+This writes one `benchmark-<model>.html` per model directory found under `results/`, plus a `benchmark-index.html` wrapper. Open `benchmark-index.html` in a browser to tab between all models in one page — each model's report loads on demand when its tab is clicked.
+
+If reports from previous rounds are already present in the directory, `benchmark-index.html` will include them alongside any newly generated ones, so the index always covers every available model regardless of which models were in the current run.
+
+Each per-model report shows attention backend results side by side, with tabs for each input sequence length, color-coded best/worst values per metric, and percentage deltas relative to the default backend.
+
+**MoE backend sweep reports** are generated automatically alongside the attention sweep reports. For each model directory that contains `moe-*` result subdirectories, `gen_report.py` writes a `moe-benchmark-<model>.html` file. A `moe-benchmark-index.html` landing page is also produced, covering all models with MoE sweep data. Open it in a browser to tab between models — same layout and features as the attention backend report.
+
 ## Agents
 
 `CLAUDE.md` has conventions for AI agents working in this repo: smoke-testing changes, launching Buildkite builds for a chosen branch/commit, and the AI-assistance disclosure rule for PRs and commits.