From ceba96df377a8367454a7f759d0e01c39d3dc713 Mon Sep 17 00:00:00 2001 From: khluu Date: Thu, 28 May 2026 17:33:55 -0700 Subject: [PATCH 1/6] Install pytest in container to fix cupy/torch.compile import failure Some vLLM release images don't include pytest, but cupy.testing._random imports it unconditionally. When torch.compile traces through cupy modules during model loading, this causes a ModuleNotFoundError that crashes the server. Installing pytest after container startup fixes this. Co-Authored-By: Claude Opus 4.6 (1M context) --- lib/server.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/server.sh b/lib/server.sh index c80fdbf..2d8ce49 100644 --- a/lib/server.sh +++ b/lib/server.sh @@ -55,6 +55,9 @@ start_server() { "$image" \ "$model" --port "$port" $serve_args + # Install pytest to avoid cupy.testing import failure during torch.compile + docker exec "$container" pip install -q pytest 2>/dev/null || true + echo "--- :memo: streaming vllm logs" ( docker logs -f "$container" 2>&1 | stdbuf -oL -eL sed 's/^/[vllm] /' ) & VLLM_LOGS_PID=$! From 5a7cf6eea8d4354a35e4ac008ca1d61607c62e4d Mon Sep 17 00:00:00 2001 From: khluu Date: Wed, 17 Jun 2026 17:50:55 -0700 Subject: [PATCH 2/6] Align deepseek_v4_flash-b200 serve args with vLLM recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Switch the DeepSeek-V4-Flash B200 workload to the published vLLM recipe serve args: TP=8 + expert parallel (was TP=2 × DP=4), and drop the --max-model-len 32768 and MTP speculative-decode flags that aren't part of the recipe. deep_gemm_mega_moe MoE backend and fp4 indexer cache are retained. Co-Authored-By: Claude Opus 4.8 (1M context) --- workloads/deepseek_v4_flash_b200.yaml | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/workloads/deepseek_v4_flash_b200.yaml b/workloads/deepseek_v4_flash_b200.yaml index 694ab37..1bcf02b 100644 --- a/workloads/deepseek_v4_flash_b200.yaml +++ b/workloads/deepseek_v4_flash_b200.yaml @@ -1,4 +1,4 @@ -# DeepSeek-V4-Flash on B200 (TP=2 × DP=4 + EP, deep_gemm_mega_moe, MTP spec-decode) +# DeepSeek-V4-Flash on B200 (TP=8 + EP, deep_gemm_mega_moe) name: deepseek_v4_flash-b200 gpu: B200 num_gpus: 8 @@ -7,21 +7,17 @@ nightly: true vllm: model: deepseek-ai/DeepSeek-V4-Flash serve_args: >- - --tensor-parallel-size 2 - --data-parallel-size 4 - --enable-expert-parallel - --max-model-len 32768 + --trust-remote-code --kv-cache-dtype fp8 --block-size 256 - --moe-backend deep_gemm_mega_moe + --enable-expert-parallel + --tensor-parallel-size 8 --attention_config.use_fp4_indexer_cache=True + --moe-backend deep_gemm_mega_moe --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 - --reasoning-parser deepseek_v4 --enable-auto-tool-choice - --trust-remote-code - --speculative_config.method=mtp - --speculative_config.num_speculative_tokens=2 + --reasoning-parser deepseek_v4 bfcl: test_categories: From 7402482ecfc26812ff447e97be11f1a44100cfec Mon Sep 17 00:00:00 2001 From: khluu Date: Thu, 18 Jun 2026 01:33:50 -0700 Subject: [PATCH 3/6] =?UTF-8?q?Park=20deepseek=5Fv4=5Fflash-b200=20(nightl?= =?UTF-8?q?y:=20false)=20=E2=80=94=20broken=20cu13=20image=20toolchain?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Build 235 (and nightly build 233) fail at vLLM engine startup on B200: the release image ships a mismatched nvidia-cu13 set (nvcc/cccl vs cuda runtime headers), so the DeepSeek-V4 mHC TileLang kernel can't JIT-compile for sm_100a ("CUDA compiler and CUDA toolkit headers are incompatible"). This is an image packaging bug (cf. vllm-project/vllm#44305), not a workload issue, and there is no serve-arg/env workaround — MHCPreOp.forward_cuda hardcodes the tilelang op with enabled() pinned True, so the torch fallback is unreachable. Set nightly: false so the nightly stops going red on a known, currently unfixable-from-our-side image bug. Still triggerable via WORKLOADS; re-enable when a vLLM image with an aligned cu13 toolchain ships. Co-Authored-By: Claude Opus 4.8 (1M context) --- workloads/deepseek_v4_flash_b200.yaml | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/workloads/deepseek_v4_flash_b200.yaml b/workloads/deepseek_v4_flash_b200.yaml index 1bcf02b..8ae6ac5 100644 --- a/workloads/deepseek_v4_flash_b200.yaml +++ b/workloads/deepseek_v4_flash_b200.yaml @@ -2,7 +2,14 @@ name: deepseek_v4_flash-b200 gpu: B200 num_gpus: 8 -nightly: true +# Parked: the vLLM release image ships a mismatched nvidia-cu13 toolchain +# (nvcc/cccl vs cuda runtime headers), so the DeepSeek-V4 mHC TileLang kernel +# fails to JIT-compile at engine startup on sm_100a ("CUDA compiler and CUDA +# toolkit headers are incompatible"). There is no runtime/serve-arg workaround +# (MHCPreOp.forward_cuda hardcodes the tilelang op). Re-enable once a vLLM image +# with an aligned cu13 toolchain ships. See vllm-project/vllm#44305. +# Still triggerable explicitly via the WORKLOADS env var. +nightly: false vllm: model: deepseek-ai/DeepSeek-V4-Flash From c01870fd6ef5194ac0af5f5fac26f6ff111174d5 Mon Sep 17 00:00:00 2001 From: khluu Date: Thu, 18 Jun 2026 01:36:52 -0700 Subject: [PATCH 4/6] Add VLLM_IMAGE_CU13 image override for B200 (CUDA 13) jobs The default VLLM_IMAGE is an x86_64 CUDA 12.9 build, but Blackwell/B200 needs a CUDA 13 image (DeepSeek-V4's mHC TileLang kernels JIT-compile against the image's CUDA toolchain at startup). Add a generic per-profile image override: a GPU profile may declare `image_env: `, and when that var is set it takes precedence over VLLM_IMAGE / VLLM_COMMIT for that profile's steps only. B200 sets `image_env: VLLM_IMAGE_CU13`, so a CUDA 13 image can be routed to Blackwell while H200/AMD keep using VLLM_IMAGE. Both image-resolution paths honor it: generate_pipeline.py bakes it into the K8s pod spec, and parse_workload.py uses it in-job (deriving the recorded commit from the cu13 image tag). The var is propagated to the step env so the in-job parser records the image/commit that actually ran. Kept generic per CLAUDE.md (GPU specifics live in gpu_profiles.yaml, not the pipeline code). README updated: new optional env var + the profile `image_env` key. Co-Authored-By: Claude Opus 4.8 (1M context) --- .buildkite/generate_pipeline.py | 24 +++++++++++++++++------- README.md | 3 ++- lib/gpu_profiles.yaml | 4 ++++ lib/parse_workload.py | 17 +++++++++++++---- 4 files changed, 36 insertions(+), 12 deletions(-) diff --git a/.buildkite/generate_pipeline.py b/.buildkite/generate_pipeline.py index 82d3703..ab1768a 100644 --- a/.buildkite/generate_pipeline.py +++ b/.buildkite/generate_pipeline.py @@ -12,6 +12,12 @@ VLLM_COMMIT commit SHA → vllm/vllm-openai:nightly- (Docker Hub) BENCH_ONLY when truthy, run vllm bench configs and skip lm_eval tasks +A GPU profile may declare ``image_env: `` (e.g. B200 → VLLM_IMAGE_CU13). +When that var is set it takes precedence over VLLM_IMAGE / VLLM_COMMIT for that +profile's steps only, so a CUDA 13 image can be routed to Blackwell while other +GPUs keep the default VLLM_IMAGE. The var is also propagated to the step env so +the in-job parser records the image/commit that actually ran. + Workloads can also set ``bench_only: true`` to apply BENCH_ONLY to that step without forcing the whole build to skip lm_eval. @@ -76,8 +82,13 @@ def is_truthy(value): return str(value or "").lower() in {"1", "true", "yes"} -def resolved_image(data): +def resolved_image(data, profile): vllm = data.get("vllm") or {} + image_env = (profile or {}).get("image_env") + if image_env: + profile_image = (os.environ.get(image_env) or "").strip() + if profile_image: + return profile_image override_image = (os.environ.get("VLLM_IMAGE") or "").strip() if override_image: return override_image @@ -191,12 +202,11 @@ def make_step(path, data, profiles): "artifact_paths": ["results/**/*"], } if profile.get("server_runtime") == "native": - step["plugins"] = [b200_k8s_plugin(ecr_pull_through(resolved_image(data)), data.get("num_gpus", 1))] - step_env = { - k: os.environ[k] - for k in ("VLLM_IMAGE", "VLLM_COMMIT", "BENCH_ONLY") - if os.environ.get(k) - } + step["plugins"] = [b200_k8s_plugin(ecr_pull_through(resolved_image(data, profile)), data.get("num_gpus", 1))] + env_keys = ["VLLM_IMAGE", "VLLM_COMMIT", "BENCH_ONLY"] + if profile.get("image_env"): + env_keys.append(profile["image_env"]) + step_env = {k: os.environ[k] for k in env_keys if os.environ.get(k)} if bench_only and "BENCH_ONLY" not in step_env: step_env["BENCH_ONLY"] = "1" if step_env: diff --git a/README.md b/README.md index 93bfb1b..e872d5a 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ vllm_bench: # perf runs (optional) — fed to the perf dashboard A few things worth knowing: -- **`gpu`** must match a key in `lib/gpu_profiles.yaml`. The profile sets the Buildkite queue, default image, HF cache path, and baseline env vars. +- **`gpu`** must match a key in `lib/gpu_profiles.yaml`. The profile sets the Buildkite queue, default image, HF cache path, and baseline env vars. A profile may also set `image_env: ` to route a different image override to that GPU — B200 uses `image_env: VLLM_IMAGE_CU13` so Blackwell pulls a CUDA 13 image while everything else stays on the default `VLLM_IMAGE`. - **`nightly`** controls only the nightly schedule. Recipes with `nightly: false` (or omitted) are still triggerable explicitly via the `WORKLOADS` env var. - **`lm_eval.tasks` is a list** because each entry runs as a separate `lm_eval` invocation — `--num_fewshot` is a single global flag, so different shot counts need separate runs. Each task's results land in `results///`. - **`vllm_bench` runs first** if both blocks are present — that way perf-pipeline bugs surface quickly instead of waiting on a full lm-eval pass. @@ -103,6 +103,7 @@ The pipeline is [**`vllm/perf-eval`**](https://buildkite.com/vllm/perf-eval). Wi **Optional env vars:** - `WORKLOADS` — comma- or newline-separated list of workload paths or stems. Runs exactly those instead of the default `nightly: true` set. +- `VLLM_IMAGE_CU13` — full Docker image URI for a CUDA 13 build. B200 (Blackwell) steps use it in preference to `VLLM_IMAGE` / `VLLM_COMMIT`, because the default `VLLM_IMAGE` is an x86_64 CUDA 12.9 build; other GPUs ignore it and keep using `VLLM_IMAGE`. This is wired through the B200 profile's `image_env` key in `lib/gpu_profiles.yaml` (see below) — any profile can opt in to its own image-override var the same way. - `NIGHTLY` — set to `1` to tag every ingested row with `nightly: true`. The dashboard's `/nightly` view filters on this to pair adjacent nightly builds; only the scheduled nightly cron should set it. **Example — trigger a build from the Buildkite UI:** diff --git a/lib/gpu_profiles.yaml b/lib/gpu_profiles.yaml index 329080c..88b417c 100644 --- a/lib/gpu_profiles.yaml +++ b/lib/gpu_profiles.yaml @@ -10,6 +10,10 @@ B200: queue: b200-k8s hf_home: /mnt/shared/hf_cache server_runtime: native + # Blackwell needs a CUDA 13 image; the default VLLM_IMAGE is an x86_64 + # CUDA 12.9 build. When VLLM_IMAGE_CU13 is set it takes precedence over + # VLLM_IMAGE / VLLM_COMMIT for B200 steps only (other GPUs keep VLLM_IMAGE). + image_env: VLLM_IMAGE_CU13 env: VLLM_DEEP_GEMM_WARMUP: skip NCCL_CUMEM_HOST_ENABLE: 0 diff --git a/lib/parse_workload.py b/lib/parse_workload.py index 91d6a5a..4f16a1e 100644 --- a/lib/parse_workload.py +++ b/lib/parse_workload.py @@ -7,7 +7,8 @@ (image, model, serve_args, env, runtime), the lm_eval task list, the vllm_bench config list, and bench ingest metadata (device/tp/precision). -Image precedence: VLLM_IMAGE > VLLM_COMMIT > workload `vllm.image` > +Image precedence: a GPU profile's `image_env` var (e.g. B200 → +VLLM_IMAGE_CU13) > VLLM_IMAGE > VLLM_COMMIT > workload `vllm.image` > `vllm/vllm-openai:latest`. When BENCH_ONLY is truthy, lm_eval task names are not validated against the registry (because they will not run). """ @@ -89,8 +90,16 @@ def load_profile(gpu: str, workload_path: str) -> dict: return profiles[gpu] -def resolve_image(vllm: dict) -> tuple[str, str]: - """Pick the image and commit using VLLM_IMAGE / VLLM_COMMIT / workload.""" +def resolve_image(vllm: dict, profile: dict) -> tuple[str, str]: + """Pick the image and commit using the profile's image_env (e.g. + VLLM_IMAGE_CU13) / VLLM_IMAGE / VLLM_COMMIT / workload.""" + image_env = (profile or {}).get("image_env") + if image_env: + profile_image = (os.environ.get(image_env) or "").strip() + if profile_image: + # Commit comes from the profile image's tag, not VLLM_COMMIT, + # so the recorded commit matches the image that actually ran. + return profile_image, commit_from_image(profile_image) override_image = (os.environ.get("VLLM_IMAGE") or "").strip() override_commit = (os.environ.get("VLLM_COMMIT") or "").strip() if override_image: @@ -241,7 +250,7 @@ def main(path: str) -> None: if bfcl: validate_bfcl(bfcl, serve_args, path) - image, vllm_commit = resolve_image(vllm) + image, vllm_commit = resolve_image(vllm, profile) env = {**(profile.get("env") or {}), **(vllm.get("env") or {})} if "HF_HOME" not in env and profile.get("hf_home"): env["HF_HOME"] = profile["hf_home"] From 02b76505549324d57406d5469dac15bf1482cc64 Mon Sep 17 00:00:00 2001 From: khluu Date: Thu, 18 Jun 2026 02:29:15 -0700 Subject: [PATCH 5/6] Revert "Add VLLM_IMAGE_CU13 image override for B200 (CUDA 13) jobs" This reverts commit c01870fd6ef5194ac0af5f5fac26f6ff111174d5. --- .buildkite/generate_pipeline.py | 24 +++++++----------------- README.md | 3 +-- lib/gpu_profiles.yaml | 4 ---- lib/parse_workload.py | 17 ++++------------- 4 files changed, 12 insertions(+), 36 deletions(-) diff --git a/.buildkite/generate_pipeline.py b/.buildkite/generate_pipeline.py index ab1768a..82d3703 100644 --- a/.buildkite/generate_pipeline.py +++ b/.buildkite/generate_pipeline.py @@ -12,12 +12,6 @@ VLLM_COMMIT commit SHA → vllm/vllm-openai:nightly- (Docker Hub) BENCH_ONLY when truthy, run vllm bench configs and skip lm_eval tasks -A GPU profile may declare ``image_env: `` (e.g. B200 → VLLM_IMAGE_CU13). -When that var is set it takes precedence over VLLM_IMAGE / VLLM_COMMIT for that -profile's steps only, so a CUDA 13 image can be routed to Blackwell while other -GPUs keep the default VLLM_IMAGE. The var is also propagated to the step env so -the in-job parser records the image/commit that actually ran. - Workloads can also set ``bench_only: true`` to apply BENCH_ONLY to that step without forcing the whole build to skip lm_eval. @@ -82,13 +76,8 @@ def is_truthy(value): return str(value or "").lower() in {"1", "true", "yes"} -def resolved_image(data, profile): +def resolved_image(data): vllm = data.get("vllm") or {} - image_env = (profile or {}).get("image_env") - if image_env: - profile_image = (os.environ.get(image_env) or "").strip() - if profile_image: - return profile_image override_image = (os.environ.get("VLLM_IMAGE") or "").strip() if override_image: return override_image @@ -202,11 +191,12 @@ def make_step(path, data, profiles): "artifact_paths": ["results/**/*"], } if profile.get("server_runtime") == "native": - step["plugins"] = [b200_k8s_plugin(ecr_pull_through(resolved_image(data, profile)), data.get("num_gpus", 1))] - env_keys = ["VLLM_IMAGE", "VLLM_COMMIT", "BENCH_ONLY"] - if profile.get("image_env"): - env_keys.append(profile["image_env"]) - step_env = {k: os.environ[k] for k in env_keys if os.environ.get(k)} + step["plugins"] = [b200_k8s_plugin(ecr_pull_through(resolved_image(data)), data.get("num_gpus", 1))] + step_env = { + k: os.environ[k] + for k in ("VLLM_IMAGE", "VLLM_COMMIT", "BENCH_ONLY") + if os.environ.get(k) + } if bench_only and "BENCH_ONLY" not in step_env: step_env["BENCH_ONLY"] = "1" if step_env: diff --git a/README.md b/README.md index e872d5a..93bfb1b 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ vllm_bench: # perf runs (optional) — fed to the perf dashboard A few things worth knowing: -- **`gpu`** must match a key in `lib/gpu_profiles.yaml`. The profile sets the Buildkite queue, default image, HF cache path, and baseline env vars. A profile may also set `image_env: ` to route a different image override to that GPU — B200 uses `image_env: VLLM_IMAGE_CU13` so Blackwell pulls a CUDA 13 image while everything else stays on the default `VLLM_IMAGE`. +- **`gpu`** must match a key in `lib/gpu_profiles.yaml`. The profile sets the Buildkite queue, default image, HF cache path, and baseline env vars. - **`nightly`** controls only the nightly schedule. Recipes with `nightly: false` (or omitted) are still triggerable explicitly via the `WORKLOADS` env var. - **`lm_eval.tasks` is a list** because each entry runs as a separate `lm_eval` invocation — `--num_fewshot` is a single global flag, so different shot counts need separate runs. Each task's results land in `results///`. - **`vllm_bench` runs first** if both blocks are present — that way perf-pipeline bugs surface quickly instead of waiting on a full lm-eval pass. @@ -103,7 +103,6 @@ The pipeline is [**`vllm/perf-eval`**](https://buildkite.com/vllm/perf-eval). Wi **Optional env vars:** - `WORKLOADS` — comma- or newline-separated list of workload paths or stems. Runs exactly those instead of the default `nightly: true` set. -- `VLLM_IMAGE_CU13` — full Docker image URI for a CUDA 13 build. B200 (Blackwell) steps use it in preference to `VLLM_IMAGE` / `VLLM_COMMIT`, because the default `VLLM_IMAGE` is an x86_64 CUDA 12.9 build; other GPUs ignore it and keep using `VLLM_IMAGE`. This is wired through the B200 profile's `image_env` key in `lib/gpu_profiles.yaml` (see below) — any profile can opt in to its own image-override var the same way. - `NIGHTLY` — set to `1` to tag every ingested row with `nightly: true`. The dashboard's `/nightly` view filters on this to pair adjacent nightly builds; only the scheduled nightly cron should set it. **Example — trigger a build from the Buildkite UI:** diff --git a/lib/gpu_profiles.yaml b/lib/gpu_profiles.yaml index 88b417c..329080c 100644 --- a/lib/gpu_profiles.yaml +++ b/lib/gpu_profiles.yaml @@ -10,10 +10,6 @@ B200: queue: b200-k8s hf_home: /mnt/shared/hf_cache server_runtime: native - # Blackwell needs a CUDA 13 image; the default VLLM_IMAGE is an x86_64 - # CUDA 12.9 build. When VLLM_IMAGE_CU13 is set it takes precedence over - # VLLM_IMAGE / VLLM_COMMIT for B200 steps only (other GPUs keep VLLM_IMAGE). - image_env: VLLM_IMAGE_CU13 env: VLLM_DEEP_GEMM_WARMUP: skip NCCL_CUMEM_HOST_ENABLE: 0 diff --git a/lib/parse_workload.py b/lib/parse_workload.py index 4f16a1e..91d6a5a 100644 --- a/lib/parse_workload.py +++ b/lib/parse_workload.py @@ -7,8 +7,7 @@ (image, model, serve_args, env, runtime), the lm_eval task list, the vllm_bench config list, and bench ingest metadata (device/tp/precision). -Image precedence: a GPU profile's `image_env` var (e.g. B200 → -VLLM_IMAGE_CU13) > VLLM_IMAGE > VLLM_COMMIT > workload `vllm.image` > +Image precedence: VLLM_IMAGE > VLLM_COMMIT > workload `vllm.image` > `vllm/vllm-openai:latest`. When BENCH_ONLY is truthy, lm_eval task names are not validated against the registry (because they will not run). """ @@ -90,16 +89,8 @@ def load_profile(gpu: str, workload_path: str) -> dict: return profiles[gpu] -def resolve_image(vllm: dict, profile: dict) -> tuple[str, str]: - """Pick the image and commit using the profile's image_env (e.g. - VLLM_IMAGE_CU13) / VLLM_IMAGE / VLLM_COMMIT / workload.""" - image_env = (profile or {}).get("image_env") - if image_env: - profile_image = (os.environ.get(image_env) or "").strip() - if profile_image: - # Commit comes from the profile image's tag, not VLLM_COMMIT, - # so the recorded commit matches the image that actually ran. - return profile_image, commit_from_image(profile_image) +def resolve_image(vllm: dict) -> tuple[str, str]: + """Pick the image and commit using VLLM_IMAGE / VLLM_COMMIT / workload.""" override_image = (os.environ.get("VLLM_IMAGE") or "").strip() override_commit = (os.environ.get("VLLM_COMMIT") or "").strip() if override_image: @@ -250,7 +241,7 @@ def main(path: str) -> None: if bfcl: validate_bfcl(bfcl, serve_args, path) - image, vllm_commit = resolve_image(vllm, profile) + image, vllm_commit = resolve_image(vllm) env = {**(profile.get("env") or {}), **(vllm.get("env") or {})} if "HF_HOME" not in env and profile.get("hf_home"): env["HF_HOME"] = profile["hf_home"] From d63a712982d3f04fe3b714603ad9837e38408303 Mon Sep 17 00:00:00 2001 From: khluu Date: Thu, 18 Jun 2026 02:29:45 -0700 Subject: [PATCH 6/6] Fix deepseek_v4_flash-b200 cu13 JIT via CCCL_DISABLE_CTK_COMPATIBILITY_CHECK The B200 failure isn't a CUDA-major problem (the image is already CUDA 13). The image's nvidia-cu13 wheels are minor-skewed (nvcc minor != cudart-headers minor), so the DeepSeek-V4 mHC TileLang kernel's sm_100a JIT compile trips CCCL's nvcc-vs-headers guard in cuda_toolkit.h ("CUDA compiler and CUDA toolkit headers are incompatible"). CCCL ships an official opt-out for exactly this ("use a newer CTK than the compiler ships"): define CCCL_DISABLE_CTK_COMPATIBILITY_CHECK. nvcc honors NVCC_APPEND_FLAGS and TileLang's compile subprocess inherits os.environ on Linux, so set NVCC_APPEND_FLAGS=-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK in the workload env to skip the (minor-only, same CUDA 13 major) check. Un-park the workload (nightly: true) now that it has a real fix. Drop the env override once the image ships version-aligned cu13 wheels. See vllm-project/vllm#44305. Co-Authored-By: Claude Opus 4.8 (1M context) --- workloads/deepseek_v4_flash_b200.yaml | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/workloads/deepseek_v4_flash_b200.yaml b/workloads/deepseek_v4_flash_b200.yaml index 8ae6ac5..8718e8a 100644 --- a/workloads/deepseek_v4_flash_b200.yaml +++ b/workloads/deepseek_v4_flash_b200.yaml @@ -2,17 +2,20 @@ name: deepseek_v4_flash-b200 gpu: B200 num_gpus: 8 -# Parked: the vLLM release image ships a mismatched nvidia-cu13 toolchain -# (nvcc/cccl vs cuda runtime headers), so the DeepSeek-V4 mHC TileLang kernel -# fails to JIT-compile at engine startup on sm_100a ("CUDA compiler and CUDA -# toolkit headers are incompatible"). There is no runtime/serve-arg workaround -# (MHCPreOp.forward_cuda hardcodes the tilelang op). Re-enable once a vLLM image -# with an aligned cu13 toolchain ships. See vllm-project/vllm#44305. -# Still triggerable explicitly via the WORKLOADS env var. -nightly: false +nightly: true vllm: model: deepseek-ai/DeepSeek-V4-Flash + env: + # The vLLM release image's nvidia-cu13 wheels are minor-skewed (nvcc minor + # != cudart-headers minor, both CUDA 13), so the DeepSeek-V4 mHC TileLang + # kernel's sm_100a JIT compile trips CCCL's nvcc-vs-headers guard ("CUDA + # compiler and CUDA toolkit headers are incompatible"). nvcc honors + # NVCC_APPEND_FLAGS and TileLang's compile subprocess inherits the env, so + # define CCCL's official opt-out to skip the (same-major) check. + # Drop this once the image ships version-aligned cu13 wheels. + # See vllm-project/vllm#44305. + NVCC_APPEND_FLAGS: "-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK" serve_args: >- --trust-remote-code --kv-cache-dtype fp8