[TEST — DO NOT MERGE] verify #1667 interp GPU lane runs on the #1622 recipe#1670
Draft
polinabinder1 wants to merge 18 commits into
Draft
[TEST — DO NOT MERGE] verify #1667 interp GPU lane runs on the #1622 recipe#1670polinabinder1 wants to merge 18 commits into
polinabinder1 wants to merge 18 commits into
Conversation
…d layout) Re-lands #1622 onto the post-#1633 top-level layout (interpretability/sparse_autoencoders/). A prior `merge main` had orphaned all of #1622's added files at the old bionemo-recipes/ path while the rest of the tree migrated; this replays the net delta at the correct paths. - Evo2SAE inference engine + steering hook (src/evo2_sae/, sae/src/sae/steering.py) - dedicated path-gated GPU CI lane + .ci_build.sh (reuses evo2_megatron's build, no fork) + .ci_test_env.sh - conftest 1B-load fixture (bionemo_load -> run_nemo2_to_mbridge + synth tiny SAE, memory-gated) - engine unit tests (test_core.py, CPU) + steering/encode GPU tests (test_steering.py) Validated on the 1B: test_core 7 passed (CPU), test_steering 9 passed (GPU). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
…nitize - non-finite strength (NaN / ±inf) is capped to a finite |value| <= MAX_CLAMP_STRENGTH, never propagated (a non-finite clamp would NaN the logits -> CUDA device-assert -> wedge) - duplicate feature_id collapses last-wins; missing strength defaults to 1.0; strength 0 kept Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
…ode_batch - _sanitize_steering coerces a negative top_k to 0 (invalid sampler arg, same class of device-fault guard as the range/clamp/temperature checks) + CPU test - encode_batch GPU test now includes an empty sequence (-> [0, n_features]); restores the empty-record branch coverage lost when the mocked test was dropped Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
Thin, non-forking image for running the engine/tests: it delegates to evo2_megatron's build via .ci_build.sh (no reimplementation of the pinned megatron stack), then installs sae + this recipe. Matches the other recipes (all of which ship a Dockerfile) and gives coworkers a turnkey 'docker build + docker run' path instead of a manual megatron setup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
- clamp_hook hardcoded torch.topk(codes, sae.top_k) + re-derived relu+topk by hand: it would AttributeError on a non-TopK SAE and could silently diverge from the SAE's true encoding. It now calls the SAE's canonical encode()/decode() (encode_pre_act only for the normalization info decode needs). - _load_sae could select ReLUSAE; it now only ever loads TopKSAE, so a non-TopK SAE can never reach the hook. - Dropped ReLU from the recipe train.py (--model-type/--l1-coeff) + 7b.sh, and the ReLU _load_sae test. (Shared sae-lib ReLUSAE untouched — esm2/codonfm use it.) Validated: 9 passed CPU (sae steering mechanism + recipe plumbing), 12 passed 1B GPU (steering). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
…core) The recipes migration renamed the NGC loader bionemo.core.data.load -> bionemo.common.data.load (matching evo2_megatron's own conftest). The old import failed on the migrated layout, so the 1B fixture hit its ImportError guard and the GPU tests SKIPPED in CI (14 passed, 5 skipped) and in the Docker image. Now imports bionemo.common with a bionemo.core fallback for older builds. Validated locally via the full auto-build path (EVO2_CKPT_DIR unset): bionemo_load fetch + run_nemo2_to_mbridge convert + GPU encode/generate -> 2 passed (126s real setup), no skip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
…ject The recipe relied on the repo-root pytest.ini to register @pytest.mark.slow, but that file isn't in the CI sparse-checkout / Docker build context, so the GPU tests raised PytestUnknownMarkWarning there (and would fail under --strict-markers). Register it locally so the recipe is self-contained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
…ocker layers) The Dockerfile copied the whole sparse_autoencoders tree before the single `RUN .ci_build.sh`, so editing any SAE file invalidated that layer and re-ran the ~30-min megatron compile. Split into two layers via a phase arg on .ci_build.sh (env | install | all): - Layer 1: COPY recipes/evo2_megatron + the recipe's .ci_build.sh, `RUN .ci_build.sh env` — the expensive mbridge build, depends ONLY on evo2_megatron, so it stays cached across code edits. - Layer 2: COPY the SAE source, `RUN .ci_build.sh install` — the two editable pip installs (seconds). A code change now invalidates only this layer. CI still calls `.ci_build.sh` (no arg = all). Adds a per-Dockerfile .dockerignore to keep node_modules/dist/__pycache__/.venv out of the build context. Not docker-built here (no docker in this env) — bash syntax + phase routing verified; CI builds it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
Loading an SAE whose input_dim != the model's hidden size (wrong SAE/model pairing) used to
succeed at load and then fail with a cryptic matmul shape error on the first encode. load() now
checks it up front and raises a clear message ("SAE input_dim=X does not match the Evo2 hidden
size=H at layer L — wrong SAE/model pairing (check --sae-ckpt-path / --layer)").
- _model_hidden_size(): read it from the model config (cheap) or a 1-token forward (ground
truth); None if neither works -> check skipped, never blocks an otherwise-fine load.
- _check_dim(): pure, unit-tested on CPU (test_check_dim_rejects_sae_model_mismatch).
NOT detectable, documented in code: a *wrong layer number* with the same hidden size — encode
still matches dims and silently yields out-of-distribution features. The SAE checkpoint records
no training layer; /health surfaces the configured layer. Follow-up: stamp the layer into the SAE
checkpoint at train time and assert it here.
Validated in the evo2_megatron venv: CPU test_core 7 passed, GPU test_steering 12 passed on the 1B
(real load() exercises the new check; 1920==1920, no false positive).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Polina Binder <pbinder@nvidia.com>
Sort encode_batch work by token length so each micro-batch holds similar-length sequences (less wasted padding on mixed-length inputs). Results are written back by original index, so the returned order still matches the input order. Add a CPU test that stubs the model and asserts input-order output despite the internal length-sort. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
Drop the separate truncated encode model (load_model_to_layer(full=False)) and serve both
paths from the one inference engine built by setup_inference_engine:
* load() builds the engine and takes self.model = unwrap_model(comp.model) + comp.tokenizer.
* encode/highlight (_forward_hidden) runs a normal full-sequence forward on that model and
reads layer L off a forward hook — the engine model is post_process=True so output_embeddings
can't be used; the hook captures the same [S,B,H] module output the steering clamp_hook reads,
so encode and steer see identical activations.
* generate() steers on self.model.decoder.layers[L] (the same module encode reads).
Removes the ~1.8x model duplication (one set of weights instead of truncated + full). The
num-microbatches double-init teardown is now just defensive (only one model inits it).
Test: add GPU test_highlight_steer_interleaving_no_bleed — encode is bit-identical across a
steered generate, and a baseline generate is unaffected by prior encode/steer history (proves
no state bleed between the shared model's highlight forward and decode path).
Validated end-to-end on the 1B-8k-bf16 (21/21 tests pass, incl. the interleaving + steering
GPU tests). 7B fidelity still unconfirmed (no 7B checkpoint available) — HOLD push for that gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Polina Binder <pbinder@nvidia.com>
Replace the 'slow' marker with an explicit @pytest.mark.skipif(not torch.cuda.is_available()) on the GPU/integration tests in test_steering.py (shared 'requires_gpu' decorator). They run when a GPU is present (CI's L4 + megatron env) and skip with a clear 'requires a GPU' reason otherwise — the conftest fixtures still further skip on too-little GPU memory or an unfetchable/unimportable checkpoint. Remove the now-unused 'slow' marker registration from pyproject. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
Remove .github/workflows/unit-tests-interpretability-recipes.yaml per review (recoverable from history / 9bedf2b). Keep .ci_build.sh + .ci_test_env.sh + the tests — those are the build/run machinery (used by the Dockerfile and manual runs), not the CI lane. How to build + run the tests is documented in the PR description; CI should later fold into the repo-wide recipe lane rather than a bespoke workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Polina Binder <pbinder@nvidia.com>
## Summary FastAPI **server + CLI** over the Evo2SAE engine (#1622). Thin wrappers — all model work lives in `core.py` — plus the input validation, resource governance, and recovery needed for a shared backend (runs behind NVIDIA SSO on Brev, reachable by many users). API routes live under **`/api`**, and the server can mount a prebuilt front-end at `/`, so the dashboard (#1623) and the API can be served from **one origin / one container**. **Rebased onto the single-engine #1622** (one inference engine serves both encode and generate; new top-level layout `interpretability/sparse_autoencoders/…`). ## Contents (new layout) - `…/src/evo2_sae/server.py` — `/api/health`, `/api/features`, `/api/annotate`, `/api/generate` (+ optional static-frontend mount at `/`) - `…/src/evo2_sae/cli.py` — `serve` / `encode` / `batch` / `generate` - `…/scripts/launch_inference.sh`; CPU contract tests `tests/test_cli.py`, `tests/test_server.py` + the shared `FakeEngine` appended to #1622's `tests/conftest.py` ## Shared logic (CLI ⇄ server live in `core`) - **`core.annotate(engine, …)`** — clean → resolve-tag → encode → tag-len, behind both CLI `encode` and server `/api/annotate`. - **`core.parse_clamp_spec(spec)`** — one parser for clamps as CLI `"ID[:STRENGTH]"` strings or server `FeatureClamp` JSON; fed in front of #1622's `_sanitize_steering` so both surfaces validate identically. ## Single-origin serving (`/api` + optional static mount) - API routes are grouped under **`/api`** (one `APIRouter` + `include_router`). - `build_app(engine, static_dir=None)` mounts a prebuilt front-end at `/` via `StaticFiles(html=True)` when `static_dir` (or the `DASHBOARD_DIST` env) points at a real directory; otherwise the server is **API-only** and `/` 404s (never crashes). The mount is generic — it serves whatever dir it's pointed at and knows nothing about the dashboard; #1623 supplies the dir + the Docker build that produces it. - This is what lets a single container serve UI + API on one port. Dev hits the same `/api/*` paths (the Vite proxy forwards `/api` without rewriting), so there's no dev/prod path drift. ## Reliability & governance - **`/api/health` 503 until ready** so readiness probes don't route to a still-loading pod; a startup load failure is caught and leaves the engine not-ready (503) rather than crashing. - **Length limits** — `/api/annotate` and `/api/generate` reject input longer than `max_seq_len` (**413**) instead of silently truncating (which would misalign the per-base `activations`/`bases` the viz plots). Generation length is otherwise auto-capped to the remaining context (no fixed token cap). - **Pick-id validation** — `/api/annotate` `mode=pick` range-checks user-supplied `feature_ids` → **400** (an out-of-range id would otherwise 500 on `IndexError`, a negative one would silently return the wrong feature). - **Steering sanitation** — out-of-range ids, extreme/non-finite strengths, `temperature<=0`, negative `top_k` are all rejected/coerced before the GPU (`_sanitize_steering`). - **CUDA-wedge recovery** — a device-side assert poisons the process's CUDA context (unrecoverable in-process). Not client-inducible (sanitation covers the reachable triggers — purely defensive), but if it happens `generate()` flips the engine not-ready (→ 503) and, when `EXIT_ON_CUDA_WEDGE=1` (set by `serve`), exits the worker so any restart-on-exit supervisor respawns it — host-independent recovery. - **Signal-safe serve** — `launch_inference.sh serve` runs the worker in the background, forwards `SIGTERM`/`SIGINT` (uvicorn graceful shutdown) before respawning, with a retry cap + backoff, so `docker stop`/k8s shuts down cleanly instead of orphaning the worker. - **Request body-size limit** (`MAX_BODY_BYTES`, default 16 MiB) → 413 — advisory (trusts `Content-Length`). - **Bounded concurrency** — Starlette's sync-endpoint threadpool capped (`MAX_CONCURRENCY`, default 8); the engine lock already serializes the single GPU. ## Architectural decisions - **Two layers: engine vs. surface.** All model work stays in `core.Evo2SAE` (#1622); `server.py`/`cli.py` are thin and share `core.annotate` + `core.parse_clamp_spec`, so the HTTP API and the CLI can't drift and there's one validated path. - **FastAPI, not raw/Flask.** We get pydantic structural validation + an async threadpool we can bound (`MAX_CONCURRENCY`) for almost no code; the domain validation that matters (`_sanitize_steering`, pick-id range) is manual either way. Raw Python would hand-roll routing/validation/concurrency; Flask would add the threadpool governance by hand. - **No app-level auth.** Deployed behind NVIDIA SSO on Brev; auth is the proxy's job, not duplicated here (CORS removed too — calls are same-origin). - **Single GPU, serialized.** The engine lock + bounded threadpool match one GPU; data-parallel replicas behind a balancer are a deferred follow-up (touches no engine code). - **`/api` prefix + generic static mount** (above) so one origin/container can serve both UI and API. ## How to run Run **inside the evo2_megatron venv** (provides `bionemo.evo2` + megatron); in the Docker image it's already active. Full dashboard run modes are in #1623's `feature_explorer/README.md`. ```bash export EVO2_CKPT_DIR=<mbridge> SAE_CKPT_PATH=<sae.pt> export FEATURE_ANNOTATIONS=<feature_metadata.parquet> EMBEDDING_LAYER=26 scripts/launch_inference.sh serve # API on :8001 (+ UI at / if DASHBOARD_DIST set) scripts/launch_inference.sh encode --sequence ATGC... # one sequence -> top features (JSON) scripts/launch_inference.sh batch --fasta in.fa --out out.parquet # many -> parquet scripts/launch_inference.sh generate --prompt ATGC... --clamp 29244:300 # steered generation ``` Tunables (env): `MAX_BODY_BYTES`, `MAX_CONCURRENCY`, `MAX_SEQ_LEN`, `PORT`, `EXIT_ON_CUDA_WEDGE`, `DASHBOARD_DIST`. ## Tests No dedicated CI lane (deferred — see #1622). Run them via the recipe's build script: ```bash cd interpretability/sparse_autoencoders/recipes/evo2 bash .ci_build.sh && source .ci_test_env.sh pytest tests/ ``` - **CPU (no model):** `test_cli.py` + `test_server.py` (FastAPI `TestClient` + `FakeEngine`: response shapes, 400/413/503, pick out-of-range → 400, `/api/generate` too-long → 413, body-size, k-bounds, clamp validation, **static-frontend mount: SPA at `/`, asset served, API reachable under `/api`, unknown `/api/*` → 404, API-only when no frontend**), plus #1622's `test_core.py` + `test_steering.py` sanitize guards. - **GPU:** `test_steering.py` — encode, in-distribution generation, steering changes the continuation, batched/empty encode, max-clamp finite, **highlight↔steer interleaving** (single-engine state-bleed check). Gated by `@pytest.mark.skipif(not torch.cuda.is_available())` — runs on a GPU box, skips otherwise. Validated on the 1B; the single-engine backend also serves the **7B at layer 26** live. ## Deferred follow-up Multi-GPU **data-parallel replicas** (one worker per GPU behind a `least_conn` balancer) for concurrent throughput — touches no engine code; left until concurrency is an observed need. **Stacked on #1622.** The dashboard (#1623) builds on this. --------- Signed-off-by: Polina Binder <pbinder@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the last hardcoded /data/interp path from the recipe scripts — WORK_ROOT now fails fast with a helpful message instead of defaulting to a machine-specific dir, matching launch_inference.sh's required-env-var style. Addresses the environment-hygiene review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Collaborator
Author
|
/ok to test 5ee24e3 |
Collaborator
Author
|
/build-ci |
Collaborator
Author
|
/ok to test 71d1be5d3e4ff3720f00a0a1e110282c4030682 |
Relocates the delta-clamp steering hook out of the shared sae library and into the recipe, so this PR's code is entirely under recipes/evo2/: sae/src/sae/steering.py -> recipes/evo2/src/evo2_sae/steering.py sae/tests/test_steering.py -> recipes/evo2/tests/test_clamp.py core.py + the moved test import from evo2_sae.steering. The recipe still consumes the shared sae lib for TopKSAE + sae.eval (unchanged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>
polinabinder1
added a commit
that referenced
this pull request
Jul 1, 2026
- Drop the workflow-level `paths:` filter: it never matched the blossom mirror push, so the lane never triggered (only the repo-wide lane fired). - Move relevance-gating into the `gate` job for push (not just merge_group): diff vs origin/main so the L4 only spins up when relevant files changed. - Scope that diff to the evo2 SAE recipe + shared `sae` lib + evo2_megatron (not all of interpretability/**, which includes codonfm/esm2). - schedule (nightly) always runs the full suite. Validated via the #1670 test PR (L4 ran: 54 recipe + 51 sae-lib tests passed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>
polinabinder1
force-pushed
the
pbinder/test-interp-gpu-lane
branch
from
571d1be to
3f0b92b
Compare
polinabinder1
force-pushed
the
pbinder/test-interp-gpu-lane
branch
from
3f0b92b to
98cce84
Compare
#1667 matrix Uses the finalized #1667 lane (changed-dirs + per-recipe matrix over interpretability/sparse_autoencoders/recipes/*). On this PR the diff touches recipes/evo2/, so the matrix = [evo2] and the L4 runs its .ci_build.sh + pytest tests/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>
polinabinder1
force-pushed
the
pbinder/test-interp-gpu-lane
branch
from
98cce84 to
db3b0b1
Compare
Collaborator
Author
|
/ok to test db3b0b1 |
polinabinder1
added a commit
that referenced
this pull request
Jul 3, 2026
The eligibility loop used `[ -f .ci_build.sh ] && echo`; when the last recipe dir (evo2, alphabetically) lacks a .ci_build.sh the short-circuit leaves exit 1, and `set -e` turns that into a job failure instead of an empty (green no-op) matrix. This bit the lane on its own branch — before #1622 lands, NO recipe has a .ci_build.sh, so ALL=[] and changed-dirs errored (the "nothing eligible" path was never exercised; #1670 validated it stacked with #1622, where evo2 IS eligible). Use `if`, which returns 0 when the body is skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: polinabinder1 <pbinder@nvidia.com>
pstjohn
pushed a commit
to pstjohn/bionemo-framework
that referenced
this pull request
Jul 7, 2026
…esence-guarded) (NVIDIA-BioNeMo#1667) ## What A GPU CI **matrix lane** for the SAE interpretability recipes under `interpretability/sparse_autoencoders/recipes/*`. Modeled on the repo-wide `unit-tests-recipes.yml`: a cheap ubuntu `changed-dirs` job discovers which interp recipes to test and emits a matrix; a per-recipe `unit-tests` job runs each on an L4 (its own `.ci_build.sh` + `pytest tests/`). ## Which recipes are eligible Any interp recipe that has its own **`.ci_build.sh`**. Today that's **evo2**. `codonfm`/`esm2` have no `.ci_build.sh`/tests yet, so they're **skipped until they add them** — at which point they auto-join this lane, no workflow change needed. (This is also the **presence guard**: before the evo2 SAE recipe lands, NVIDIA-BioNeMo#1622, nothing is eligible → the whole lane is a green no-op, so it can merge first.) ## What runs, when | What the PR changes | Which recipes run | |---|---| | a recipe's own dir — `…/recipes/<X>/**` | just **`<X>`** | | the shared `sae` lib — `…/sparse_autoencoders/sae/**` | **all** eligible recipes (they all depend on sae) | | this workflow file | **all** eligible recipes | | **nightly** `schedule` @ 09:00 UTC | **all** eligible recipes | | anything else (other recipes' dirs, docs, …) | **none** — empty matrix, green no-op | - **Megatron *build* is per-recipe — only evo2 builds it.** Each recipe's own `.ci_build.sh` owns its build: **evo2** compiles/installs the mbridge `bionemo.evo2` (megatron) stack; a future HF-native **esm2** or custom **codonfm** builds its own thing, *no megatron*. So a codonfm/esm2 change never pays for the Evo2 megatron build — it runs only on an evo2 change (or an `sae`/nightly run, which tests every consumer, evo2 included). - **Container *image* is currently shared.** All matrix entries use one base image (`svcbionemo023/…pytorch26.04-py3-squashed`, dockerhub-cached → cheap after first pull) — the megatron-capable base evo2 needs. A future HF-native recipe *runs in* that base but doesn't build megatron on top. If a recipe later wants a lighter image, the matrix can carry a **per-recipe `image`** (like the repo-wide lane's `matrix.recipe.image`) — not needed while evo2 is the only active recipe. Net: **shared base image, per-recipe (megatron-or-not) build.** - Each eligible recipe runs on the **L4** against its **CI-sized model** (evo2 = the auto-built **1B**). The **7B/L26** path is manual (`EVO2_CKPT_DIR`), never in CI. - Not covered: the React dashboard frontend (no JS build step) and offline `extract`/`train` scripts (no unit tests). ## Validation Proven end-to-end on a real L4 via the NVIDIA-BioNeMo#1670 test PR: **https://github.com/NVIDIA-BioNeMo/bionemo-recipes/actions/runs/28535969801** — the L4 built via `.ci_build.sh` and ran the evo2 recipe suite green (**54 recipe tests passed**). The `changed-dirs` matrix logic was verified locally across 8 scenarios (per-recipe / sae-change / nightly / pre-NVIDIA-BioNeMo#1622-empty). Trigger = a maintainer commenting `/ok to test <sha>`. --------- Signed-off-by: polinabinder1 <pbinder@nvidia.com> Co-authored-by: root <root@nvidia-lepton040.cm.cluster> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Throwaway test PR — do not merge. Proof that #1667's interp-recipes GPU matrix lane runs on a real L4.
Current state
Branch = #1622 (steering hook moved into the recipe) + the final genericized #1667 workflow — a
changed-dirs→ per-recipe matrix overinterpretability/sparse_autoencoders/recipes/*(eligible = has.ci_build.sh; today only evo2; codonfm/esm2 auto-join when they add one).✅ Result — the lane ran green on a real L4
Run: https://github.com/NVIDIA-BioNeMo/bionemo-recipes/actions/runs/28545830763
changed-dirs→Recipes to run: ["interpretability/sparse_autoencoders/recipes/evo2"]— the matrix correctly resolved to[evo2](validates the discovery logic; codonfm/esm2 correctly absent — no.ci_build.sh).interp-unit-tests (evo2)→ completed/success on an NVIDIA L4 (.ci_build.shbuild →pytest tests/).This validates the whole generic-matrix machinery end-to-end: discovery → matrix → per-recipe L4 build + test.
What this test established (and the bug it caught)
[evo2]; the earlier workflow-levelpaths:filter silently blocked the lane from triggering at all under the blossom mirror push — removed, with relevance now decided inchanged-dirs(diff vsmain).recipes/evo2/+recipes/evo2_megatron/; asae/or codonfm/esm2 change doesn't drag anyone through the Evo2 build.Runs on the L4 / 1B; the 7B path is manual (
EVO2_CKPT_DIR), never in CI. Trigger = a maintainer commenting/ok to test <sha>. Refs: #1667 (the lane), #1622 (the recipe).