Add hyenaND-retrofit Claude skill#115
Merged
Merged
Conversation
Adds a Claude Code / Cowork skill that automates the attention -> HyenaND swap. Targets two paths: - Native: in-repo configs that use the build_attention_net / build_hyena_net / build_hybrid_net builders. Swap is mechanical (change the builder call, flip compile_compatible_fftconv, pick a layer pattern for hybrids). - Foreign: external PyTorch models (timm ViT, HF transformers, hand- written) that use nn.MultiheadAttention or F.scaled_dot_product_ attention. Skill emits a sibling file with a full Hyena module constructed from paper-grounded vision/medical/genomics/PDE defaults. Paper-grounded defaults live in references/defaults.md (omega_0, register count, Gaussian mask, FFT padding, RoPE on/off by modality, plus the hybrid layer patterns from sec 5.2/5.3/5.5: HHHA x3 for ImageNet, HHAA for SwinUNETR, H_2 mixing for Evo2-style genomics LMs). Three eval cases under evals/ exercise the native pure-swap, native hybrid, and foreign-timm-vit paths. Iteration 1 scored 23/23 with the skill vs 20/22 without (baseline missed __main__ smoke-test blocks); foreign-path assertions tied -- a known gap to tighten next iteration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Farhad Ramezanghorbani <farhad.ghorbani@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tshimko-nv
approved these changes
May 29, 2026
tshimko-nv
left a comment
Collaborator
There was a problem hiding this comment.
Looks good - one small comment about Claude vs. universal skill implementation, but otherwise, should be good.
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Claude Code skill that automates the attention -> HyenaND swap. Two paths:
build_attention_net/build_hyena_net/build_hybrid_netbuilders. Swap is mechanical — change the builder call, dropcompile_compatible_fftconv = False, pick a layer pattern for hybrids. Sibling files are bare config shims (no__main__block, matching the canonicalhyena_patch16.py/hybrid_hhha.py).nn.MultiheadAttentionorF.scaled_dot_product_attention. Skill emits a sibling file with a full Hyena module plus a drop-inHyenaAttnAdapterthat bridges the three nn.MHA ↔ Hyena interface gaps: tuple return, kwarg surface (need_weights/attn_mask/key_padding_mask), and channel layout / CLS handling.The foreign path is organized around four orthogonal axes the user fixes up front:
data_dim ∈ {1, 2, 3}— picksConvNd, setsCKConvND/SIRENKernelND/GaussianModulationND.causal ∈ {True, False}— setsfft_padding,use_rope, maskparametrization,omega_0.tokens [B, N, C]vsfeature_map [B, C, *S].(out, None)tuple vs bare tensor.One parameterized adapter covers all sixteen combinations. FFT backend is selected automatically from the axes:
fft_backend="subq_ops"(optimized CUDA kernel fromsubquadratic_ops_torch) whendata_dim == 2 and not causal, elsefft_backend="torch_fft"— the latter is required for 1D, 3D, circular padding, fp16, and causal paths becausesubq_opsis 2D-only in the current nvSubquadratic wiring (the upstream package shipsfft_causal_conv1dandcausal_conv1dbut they are not exposed).A foot-guns section calls out the failures that bite in practice: INT32
im2coloverflow in large 3D short conv (≥160³ trips it; fix isDepthwiseFFTConv3d), RoPE divisibility (% 2/% 4/% 6for 1D / 2D / 3D, must hold at every hierarchical stage),L_cachememory blowup ((2L-1)^D × D × 4bytes — D=3 / L=512 ≈ 12.8 GB), state_dict-prefix mismatch on pretrained-attention loading, and thesubq_opsconstraint set (asserts loudly ondata_dim ≠ 2,fft_padding ≠ "zero", causal, or fp16).Hierarchical encoders with a
use_hyena=Trueflag are an explicit exception to the sibling-file rule — the natural API is an in-place edit.Five eval cases under
evals/cover the matrix:native-pure-swap,native-hybrid— v5_patch and vit5_hybrid configs.foreign-timm-vit— 2D, tokens, with CLS.foreign-3d-feature-map— 3D, feature_map, no CLS,F.scaled_dot_product_attentionon[B, C, D, H, W].foreign-1d-causal-lm— 1D, causal tokens,attn_maskcall site.Together they exercise every (
data_dim,causal,host_layout, prefix-tokens) combination the skill is meant to handle.