[Draft]: Latest feature/nemotron-3-vl-conv3d - #2
Draft
collinmccarthy wants to merge 30 commits into
Draft
Conversation
…-project#37148) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
…ct#37200) Signed-off-by: raushan <raushan@huggingface.co>
…m-project#37090) Signed-off-by: haosdent <haosdent@gmail.com> Co-authored-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Max de Bayser <maxdebayser@gmail.com> Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
…n possible (vllm-project#36982) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
…ts (vllm-project#37215) Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
…_context_set (vllm-project#36288) Signed-off-by: Lucas Kabela <lucaskabela@meta.com>
…6915) Signed-off-by: sfeng33 <4florafeng@gmail.com> Signed-off-by: Flora Feng <4florafeng@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
vllm-project#37213) Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
…#37197) Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: zhxchen17 <zhxchen17@fb.com>
…lm-project#36688) (vllm-project#36779) Signed-off-by: Krish Gupta <krishom70@gmail.com>
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com> Signed-off-by: Elvir Crncevic <elvir@anthropic.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…models (vllm-project#35448) Signed-off-by: EdalatiAli <aliedalati@cohere.com>
…37115) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
…-project#37064) Signed-off-by: yfama <yuchengu@gmail.com>
…t#36356) Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: juliendenize <julien.denize@mistral.ai>
…project#34389) Signed-off-by: tianrengao <terrygao87@gmail.com>
Signed-off-by: Collin McCarthy <cmccarthy@nvidia.com>
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Signed-off-by: Collin McCarthy <cmccarthy@nvidia.com>
The Megatron->HF bridge in
Megatron-Bridge/src/megatron/bridge/models/nemotron_vl/nemotron_vl_bridge.py
has no `llava_model.vision_model.decoder.layers.*.ls{1,2}` mapping, so the
exported HF checkpoint carries no `ls1`/`ls2` keys. That is consistent
with Megatron's RADIOViTModel using a plain TransformerLayer with no
LayerScale (`ls=1` throughout).
vLLM's RADIO encoder layer (InternVisionEncoderLayer), however, does
multiply the attention and MLP residual contributions by `ls1` and
`ls2`. The constructor is supposed to seed them to ones via
`nn.Parameter(config.initializer_factor * torch.ones(self.embed_dim))`,
which works in older vLLM (e.g. 0.1.dev92 used by the omni branch) but
no longer initializes the parameter eagerly in newer vLLM (0.15.x), so
they are left at uninitialized memory (~0 garbage in practice). That
collapses every encoder layer to `x = x + ~0*attn + ~0*mlp ~= x` and
turns the entire 32-layer ViT into a near-identity pass-through. On
Nemotron Nano v3 VL during GRPO this alone accounts for ~0.7 of the
Token Mult Prob Error between vLLM-generated logprobs and Megatron-
recomputed logprobs (TMPE 1.72 -> 1.03 with this fix).
Workaround (this commit): at the end of RadioModel.load_weights, force
`ls1`/`ls2` to ones so the encoder forward matches the Megatron contract.
The existing `if suffix in {"ls1", "ls2"}: continue` skip in the load
loop is kept since the HF checkpoint legitimately has no values to load.
Longer-term, the proper fix is for Megatron-Bridge to export a synthetic
`ls1`/`ls2` = ones for every encoder layer so the HF checkpoint becomes
self-contained and the workaround can be dropped.
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
fix: force RADIO ViT LayerScale (ls1/ls2) to 1.0 after weight load
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For internal review only. See vllm-project/vllm@main...collinmccarthy:vllm:feature/nemotron-3-vl-conv3d-extra for commits.
video_temporal_patch_sizefor backwards compat. with Nemotron v2