ci: GPU matrix lane for interpretability SAE recipes (merge-first, presence-guarded) by polinabinder1 · Pull Request #1667 · NVIDIA-BioNeMo/bionemo-recipes

polinabinder1 · 2026-06-30T00:11:30Z

What

A GPU CI matrix lane for the SAE interpretability recipes under interpretability/sparse_autoencoders/recipes/*. Modeled on the repo-wide unit-tests-recipes.yml: a cheap ubuntu changed-dirs job discovers which interp recipes to test and emits a matrix; a per-recipe unit-tests job runs each on an L4 (its own .ci_build.sh + pytest tests/).

Which recipes are eligible

Any interp recipe that has its own .ci_build.sh. Today that's evo2. codonfm/esm2 have no .ci_build.sh/tests yet, so they're skipped until they add them — at which point they auto-join this lane, no workflow change needed. (This is also the presence guard: before the evo2 SAE recipe lands, #1622, nothing is eligible → the whole lane is a green no-op, so it can merge first.)

What runs, when

What the PR changes	Which recipes run
a recipe's own dir — `…/recipes/<X>/**`	just `<X>`
the shared `sae` lib — `…/sparse_autoencoders/sae/**`	all eligible recipes (they all depend on sae)
this workflow file	all eligible recipes
nightly `schedule` @ 09:00 UTC	all eligible recipes
anything else (other recipes' dirs, docs, …)	none — empty matrix, green no-op

Megatron build is per-recipe — only evo2 builds it. Each recipe's own .ci_build.sh owns its build: evo2 compiles/installs the mbridge bionemo.evo2 (megatron) stack; a future HF-native esm2 or custom codonfm builds its own thing, no megatron. So a codonfm/esm2 change never pays for the Evo2 megatron build — it runs only on an evo2 change (or an sae/nightly run, which tests every consumer, evo2 included).
Container image is currently shared. All matrix entries use one base image (svcbionemo023/…pytorch26.04-py3-squashed, dockerhub-cached → cheap after first pull) — the megatron-capable base evo2 needs. A future HF-native recipe runs in that base but doesn't build megatron on top. If a recipe later wants a lighter image, the matrix can carry a per-recipe image (like the repo-wide lane's matrix.recipe.image) — not needed while evo2 is the only active recipe. Net: shared base image, per-recipe (megatron-or-not) build.
Each eligible recipe runs on the L4 against its CI-sized model (evo2 = the auto-built 1B). The 7B/L26 path is manual (EVO2_CKPT_DIR), never in CI.
Not covered: the React dashboard frontend (no JS build step) and offline extract/train scripts (no unit tests).

Validation

Proven end-to-end on a real L4 via the #1670 test PR:
https://github.com/NVIDIA-BioNeMo/bionemo-recipes/actions/runs/28535969801 — the L4 built via .ci_build.sh and ran the evo2 recipe suite green (54 recipe tests passed). The changed-dirs matrix logic was verified locally across 8 scenarios (per-recipe / sae-change / nightly / pre-#1622-empty). Trigger = a maintainer commenting /ok to test <sha>.

…2_megatron, presence-guarded) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

copy-pr-bot · 2026-06-30T00:11:33Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

coderabbitai · 2026-06-30T00:11:37Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 62c13823-75d4-465d-8b38-22c5bd09936c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch ci/interp-evo2-gpu-lane

_{Comment @coderabbitai help to get the list of available commands.}

polinabinder1 · 2026-06-30T00:18:08Z

RFR: workflow-only GPU CI lane for the interpretability SAE recipes (evo2), modeled on the evo2_megatron lane and presence-guarded so it merges to main first as a green no-op and auto-activates once #1622 lands — verified locally on H100s (.ci_build.sh + pytest tests/ → 54 passed, 0 skipped).

polinabinder1 · 2026-06-30T00:38:32Z

/ok to test 797bdb7

Avoid summoning the L4 GPU runner when there is no real work: - Presence + relevance now live in a cheap `gate` job on ubuntu-latest; the GPU `unit-tests` job runs only when `gate` says so. - merge_group ignores `paths`, so the gate diffs against origin/main and skips the GPU suite when no interpretability/evo2 files changed - previously every merge into main rebuilt the Evo2 1B. - No-op refs (recipe absent) no longer occupy a GPU runner at all. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The lane ran only recipes/evo2/tests, leaving the shared SAE engine (sparse_autoencoders/sae) — which every recipe depends on — untested. Add a separate step running pytest on ../../sae/tests (kernels, topk, streaming, steering, and the tensor-parallel math tests; 51 tests). Separate step (not appended to the recipe run) for distinct failure attribution. .ci_build.sh already installs the SAE lib editable, and the two suites import cleanly apart (sae/tests is a package; recipe tests is not), so no shared-session basename clash. Validated on a single GPU: 51 passed in ~2min. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jwilber · 2026-07-01T01:36:46Z

HAve we verified this works with the CI-supported runners? do we have a log of that anywhere?

jstjohn · 2026-07-01T17:47:48Z

/ok to test 7b593ff

- Drop the workflow-level `paths:` filter: it never matched the blossom mirror push, so the lane never triggered (only the repo-wide lane fired). - Move relevance-gating into the `gate` job for push (not just merge_group): diff vs origin/main so the L4 only spins up when relevant files changed. - Scope that diff to the evo2 SAE recipe + shared `sae` lib + evo2_megatron (not all of interpretability/**, which includes codonfm/esm2). - schedule (nightly) always runs the full suite. Validated via the #1670 test PR (L4 ran: 54 recipe + 51 sae-lib tests passed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>

… updated #1667 workflow Re-validates the interp GPU lane after (a) moving the steering hook out of shared sae into recipes/evo2/, and (b) the #1667 trigger/relevance fix (no paths filter; gate diffs vs main, scoped to evo2 + sae + evo2_megatron). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>

…ron base) Gate on recipes/evo2/ + recipes/evo2_megatron/ only (drop the shared sae/ glob so a sae change - shared with codonfm/esm2 - doesn't drag those devs through the evo2 build). Test recipes/evo2/tests/ only (drop the separate sae/tests run; sae needs its own lane). evo2_megatron stays in the gate because the SAE engine wraps its internal bionemo.evo2 APIs, so a megatron change can break the recipe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>

… workflow Uses the finalized #1667 lane: gate on recipes/evo2/ + recipes/evo2_megatron/ (no sae), tests recipes/evo2/tests/ only. Re-validates the trigger + the steering move on a real L4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>

Replaces the single evo2 job with a changed-dirs + matrix pattern (modeled on the repo-wide unit-tests-recipes.yml), scoped to interpretability/sparse_autoencoders/recipes/*: - Eligible = any interp recipe with its own .ci_build.sh. Today that's evo2; codonfm/esm2 auto-join once they add a .ci_build.sh + tests. Empty before #1622 lands -> whole lane is a green no-op (presence guard = has .ci_build.sh). - A change under a recipe runs that recipe; a change to shared sae/, this workflow, or the nightly runs ALL eligible recipes. Each recipe's own .ci_build.sh owns its build, so a codonfm/esm2 change never triggers the Evo2 megatron build (and vice-versa). Matrix-selection logic verified locally across 8 scenarios. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>

#1667 matrix Uses the finalized #1667 lane (changed-dirs + per-recipe matrix over interpretability/sparse_autoencoders/recipes/*). On this PR the diff touches recipes/evo2/, so the matrix = [evo2] and the L4 runs its .ci_build.sh + pytest tests/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>

polinabinder1 · 2026-07-01T22:34:57Z

/ok to test db3b0b1

copy-pr-bot · 2026-07-01T22:35:00Z

/ok to test db3b0b1

@polinabinder1, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

jwilber · 2026-07-02T22:21:58Z

Reviewed with Polina in a meeting together, lgtm.

jstjohn · 2026-07-02T23:53:13Z

/ok to test bf06765

The eligibility loop used `[ -f .ci_build.sh ] && echo`; when the last recipe dir (evo2, alphabetically) lacks a .ci_build.sh the short-circuit leaves exit 1, and `set -e` turns that into a job failure instead of an empty (green no-op) matrix. This bit the lane on its own branch — before #1622 lands, NO recipe has a .ci_build.sh, so ALL=[] and changed-dirs errored (the "nothing eligible" path was never exercised; #1670 validated it stacked with #1622, where evo2 IS eligible). Use `if`, which returns 0 when the body is skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>

polinabinder1 · 2026-07-03T00:58:09Z

/ok to test 6399b96

ci(evo2-sae): add interpretability GPU unit-test lane (modeled on evo…

797bdb7

…2_megatron, presence-guarded) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

polinabinder1 marked this pull request as ready for review June 30, 2026 00:18

polinabinder1 requested review from jstjohn, pstjohn and trvachov as code owners June 30, 2026 00:18

polinabinder1 requested review from jwilber and savitha-eng June 30, 2026 00:26

polinabinder1 mentioned this pull request Jun 30, 2026

feat(evo2-sae): streaming smoke test (producer/consumer) + Lepton convergence config #1668

Open

root and others added 2 commits June 30, 2026 16:36

polinabinder1 mentioned this pull request Jul 1, 2026

[TEST — DO NOT MERGE] verify #1667 interp GPU lane runs on the #1622 recipe #1670

Draft

polinabinder1 changed the title ~~ci(evo2-sae): GPU unit-test lane for interpretability recipes (merge-first, presence-guarded)~~ ci: GPU matrix lane for interpretability SAE recipes (merge-first, presence-guarded) Jul 1, 2026

jwilber approved these changes Jul 2, 2026

View reviewed changes

jstjohn approved these changes Jul 2, 2026

View reviewed changes

polinabinder1 added this pull request to the merge queue Jul 3, 2026

Merged via the queue into main with commit fdce924 Jul 3, 2026
10 checks passed

polinabinder1 deleted the ci/interp-evo2-gpu-lane branch July 3, 2026 01:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: GPU matrix lane for interpretability SAE recipes (merge-first, presence-guarded)#1667

ci: GPU matrix lane for interpretability SAE recipes (merge-first, presence-guarded)#1667
polinabinder1 merged 7 commits into
mainfrom
ci/interp-evo2-gpu-lane

polinabinder1 commented Jun 30, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 30, 2026

Uh oh!

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading

Review skipped

Uh oh!

polinabinder1 commented Jun 30, 2026

Uh oh!

polinabinder1 commented Jun 30, 2026

Uh oh!

jwilber commented Jul 1, 2026

Uh oh!

jstjohn commented Jul 1, 2026

Uh oh!

polinabinder1 commented Jul 1, 2026

Uh oh!

copy-pr-bot Bot commented Jul 1, 2026

Uh oh!

jwilber commented Jul 2, 2026

Uh oh!

jstjohn commented Jul 2, 2026

Uh oh!

polinabinder1 commented Jul 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

polinabinder1 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Which recipes are eligible

What runs, when

Validation

Uh oh!

copy-pr-bot Bot commented Jun 30, 2026

Uh oh!

coderabbitai Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

polinabinder1 commented Jun 30, 2026

Uh oh!

polinabinder1 commented Jun 30, 2026

Uh oh!

jwilber commented Jul 1, 2026

Uh oh!

jstjohn commented Jul 1, 2026

Uh oh!

polinabinder1 commented Jul 1, 2026

Uh oh!

copy-pr-bot Bot commented Jul 1, 2026

Uh oh!

jwilber commented Jul 2, 2026

Uh oh!

jstjohn commented Jul 2, 2026

Uh oh!

polinabinder1 commented Jul 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

polinabinder1 commented Jun 30, 2026 •

edited

Loading

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading