Move GPU CI from T4 to L40S

[qgallouedec]

Our GPU jobs run on aws-g4dn-2xlarge (T4, Turing), which has no bf16 and no FlashAttention-2 — CI exercises fp16/eager paths that no user runs — and so little VRAM that we are capped at 4 test workers.

	aws-g4dn-2xlarge (before)	aws-g6e-4xlarge (after)
GPU	1× Tesla T4	1× L40S
VRAM	15 GB	45 GB
Compute capability	7.5 (Turing)	8.9 (Ada)
bf16 / FA2	✗	✓
vCPU (physical cores)	8 (4)	64 (32)
RAM	30 GB	497 GB

Multi-GPU jobs are unchanged on aws-g5-12xlarge-cache (2× A10G, 23 GB each). One fix there: slow-tests.yml's multi-GPU job requested CUDA_VISIBLE_DEVICES: "0,1" while pinned to the single-GPU T4 group, so the slow multi-GPU suite was never actually running on 2 GPUs — it now points at the same multi-GPU group as tests.yml.

Observed speedup

~5× on the same job, same test set:

Job	Before (T4, 4 workers)	After (L40S, 32 workers)
Tests (3.13)	~46 min	9m22s
Tests without optional dependencies	~28 min	6m42s

Why hardcode 32 workers?

The Makefile now pins pytest -n 32 instead of -n auto.

Pinning keeps the worker count independent of whichever runner picks up the job. 32 is what -n auto resolved to on this one.

Why 32 is the right ceiling: CI historically ran 4 workers on 15 GB, i.e. ~3.8 GB of VRAM per test. 32 workers on 45 GB is ~1.4 GB per test — tighter, not looser — and it passes today. So transient OOM is a real possibility. If we hit it, the preferred fix is to chase the few high-VRAM tests; failing that (less preferable) we drop 32 → 24 → 16.

Known failure: test_train_padding_free (SFT + DPO), marked xfail

These tests are gated on is_ampere_or_newer(), so they have always been skipped on the T4 and are only now running for the first time. They fail for a reason unrelated to the hardware: kernels-community/flash-attn2 publishes build variants for torch 2.7–2.12 only, and our env installs torch 2.13.

Marked non-strict xfail, so they will simply XPASS once the kernel builds land: huggingface/kernels-community#1082.

---

Note

Medium Risk
Changes affect all GPU CI signal and parallelism (OOM risk at 32 workers); product code is untouched aside from expected xfails on newly executed Ampere-only tests.

Overview
Moves single-GPU GPU CI from aws-g4dn-2xlarge (T4) to aws-g6e-4xlarge (L40S) across the main test workflows (tests.yml, slow-tests.yml, tests-experimental.yml, tests_latest.yml, tests_transformers_branch.yml), so CI can exercise bf16/FlashAttention-style paths closer to real Ada hardware and use more VRAM/CPU.

In slow-tests.yml, the multi-GPU job’s runner group is changed from the single-GPU T4 pool to aws-g5-12xlarge-cache, aligning it with the 2-GPU setup implied by CUDA_VISIBLE_DEVICES: "0,1".

The Makefile test target switches parallel pytest from -n auto to -n 32 to stabilize worker count on the new runners.

SFT and DPO test_train_padding_free gain a non-strict @pytest.mark.xfail because kernels-community/flash-attn2 has no torch 2.13 build yet (these tests were previously skipped on T4 via Ampere gating).

^{Reviewed by Cursor Bugbot for commit 9d4b398. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 970072b139

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1dbe172. Configure here.}