[NVBug: 6563509] Harden meta-device skeleton build; drop Phi-3-vision / Phi-4-multimodal PTQ support

CHANGELOG.rst

@@ -83,6 +83,7 @@ Changelog
 - Remove the deprecated ``examples/llm_autodeploy`` example (deprecated in 0.45). Use TensorRT-LLM's `AutoDeploy <https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/auto_deploy>`_ directly together with ModelOpt PTQ in ``examples/hf_ptq``.
 - Remove the deprecated ``examples/llm_qad`` Megatron-LM QAD example (deprecated in 0.45). Use the `megatron_bridge QAD example <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/megatron_bridge#quantization-aware-distillation-qad>`_ instead, which provides a simpler Python-based interface and better model coverage.
 - Dropped VILA / NVILA vision-language model support in ``examples/hf_ptq``. VILA's modeling code requires ``transformers<=4.50.0``, which conflicts with ModelOpt's minimum supported ``transformers`` version. The VILA-specific bootstrap (repo clone, ``requirements-vila.txt``) and loading paths in ``example_utils.py`` have been removed.
+- Dropped **Phi-3-vision** and **Phi-4-multimodal** PTQ support in ``examples/hf_ptq`` (NVBug 6563509). Their bundled remote code predates Transformers v5 and no longer loads on any version ModelOpt supports (``transformers>=4.57``): Phi-4-multimodal requires ``transformers<4.52`` because it reaches ``prepare_inputs_for_generation`` through ``peft``, which needs ``PreTrainedModel`` to still inherit ``GenerationMixin``, and both models declare ``_tied_weights_keys`` as a list, which Transformers 5.x rejects. The support-matrix row, the ``phi4mm`` model type, the multimodal-detection heuristics that only ever matched these two (``vision_lora`` / ``audio_processor`` / ``embd_layer.image_embd_layer``), the ``Phi3Image`` / ``PhiImage`` embedding-export exclusions, and the ``modelopt_recipes/huggingface/phi4mm/`` recipes have been removed. Text-only Phi-3/Phi-4 and Phi-3.5-MoE are unaffected.
 
 **Deprecations**
 
@@ -105,6 +106,7 @@ Changelog
 - Fix ``examples/vllm_serve`` serving shared experts uncalibrated: their ``gate_proj``/``up_proj`` quantizer keys were not merged into ``gate_up_proj`` on reload, so they matched no module and were dropped.
 - Fix Qwen3-VL MoE PTQ failing on ``transformers>=5.12`` with ``AttributeError: 'QuantQwen3VLMoeTextExperts' object has no attribute 'hidden_size'`` (NVBug 6518551). transformers 5.12 moved ``Qwen3VLMoeTextExperts`` onto the standard ``@use_experts_implementation`` fused layout (``hidden_size``/``expert_dim`` renamed to ``hidden_dim``/``intermediate_dim``, ``gate_up_proj`` transposed to ``(num_experts, 2*intermediate_dim, hidden_dim)``, two ``F.linear`` calls per expert), but the legacy ``_QuantQwen3VLMoeTextExperts`` wrapper stayed statically registered and shadowed on-the-fly detection. The new layout is now left to ``register_fused_experts_on_the_fly``, which claims it with the generic ``_QuantFusedExperts``; the legacy wrapper is still registered on ``transformers<5.12``, whose ``torch.bmm``-based forward the generic wrapper cannot intercept.
 - Fix ``examples/hf_ptq`` multi-node FSDP2 export (``--use_fsdp2``) failing with ``RuntimeError: Cannot set version_counter for inference tensor``. ``export_quantized`` wrapped its whole body in ``torch.inference_mode()``, so the full params gathered by ``get_model_state_dict(full_state_dict=True)`` were inference tensors and the subsequent ``state_dict()`` -> ``param.detach()`` could not set their version counter. The export context is now ``torch.no_grad()``, which still disables autograd but keeps the gathered params as normal tensors.
+- Fix ``examples/hf_ptq`` aborting the whole model load when the throwaway meta-device skeleton it builds to size ``infer_auto_device_map`` cannot be constructed (NVBug 6563509). The skeleton used ``init_empty_weights(include_buffers=True)``, which accelerate implements as a bare global ``torch.device("meta")`` context: *every* tensor built in ``__init__`` lands on meta, so remote-code checkpoints that derive scalar hyperparameters from real tensors there failed with ``Tensor.item() cannot be called on meta tensors`` before ``from_pretrained`` was ever reached. That made the probe stricter than the loader it predicts -- Transformers 4.x builds under ``include_buffers=False`` and Transformers 5.x under a meta context patched by ``meta_device_safe_creation_ops()`` -- so it could kill loads that would otherwise have succeeded. The skeleton now uses ``include_buffers=False``, which patches only ``nn.Module.register_parameter`` and leaves ``__init__`` arithmetic on a real device (module sizes are unchanged: ``compute_module_sizes`` reads shape and dtype, not storage). If it still cannot be built, the memory estimate is skipped with a warning instead of failing the run; ``--use_seq_device_map`` (which applies ``--gpu_max_mem_percentage``) and ``--batch_size`` cover the lost heuristic.
 - Fix unified HF export of multimodal models whose vision tower carries its own ``PrefixChange`` conversion (``LlavaForConditionalGeneration`` on ``transformers>=5.12`` — NVBug 6525511). transformers collects conversion mappings recursively and scopes each sub-model's transforms to that sub-module via ``scope_prefix``, matching only keys under that prefix. ModelOpt's quant-aware reverse conversion read the raw patterns and ignored ``scope_prefix``, so the vision tower's "add a ``vision_model.`` prefix" rule was applied to *every* key in the state dict: an exported llava-1.5-13b checkpoint had all 758 tensors moved under a bogus top-level ``vision_model.`` namespace (``vision_model.language_model.*``, ``vision_model.lm_head.*``), and vLLM rejected it with ``ValueError: There is no module or parameter named 'vision_model' in LlavaForConditionalGeneration``. Reverse rename rules now carry their scope and are applied only to keys under it, matching transformers' own ``WeightTransform._scoped_match`` semantics. ``Gemma3ForConditionalGeneration`` was affected identically and is fixed by the same change.
 
 0.45 (2026-07-02)

examples/hf_ptq/README.md

@@ -119,7 +119,6 @@ Please reference our [framework scripts](#framework-scripts) and our [docs](http
 | Whisper<sup>9</sup> | ✅ | ❌ | ❌ | ❌ | - |
 | Nemotron-3 | ✅ | ❌ | ❌ | ❌ | ✅ |
 | Llava (VLM)<sup>11</sup> | ✅ | ✅<sup>12</sup> | ✅ | ✅ | - |
-| Phi-3-vision, Phi-4-multimodal (VLM)<sup>11</sup> | ✅ | ✅<sup>12</sup> | ✅ | ✅ | ✅ |
 | Qwen2, 2.5-VL (VLM)<sup>11</sup> | ✅ | ✅<sup>12</sup> | ✅ | ✅ | ✅ |
 | Gemma 3 (VLM)<sup>11</sup> | ✅ | - | - | - | - |
 | Nemotron VL (VLM)<sup>11,13</sup> | ✅ | - | - | - | ✅ |

examples/hf_ptq/example_utils.py

@@ -177,12 +177,6 @@ def _is_multimodal_config(config):
     """Check if a config indicates a multimodal model (config-only version of is_multimodal_model)."""
     return (
         hasattr(config, "vision_config")  # Standard vision config (e.g., Qwen2.5-VL)
-        or getattr(config, "model_type", "") == "phi4mm"  # Phi-4 multimodal
-        or hasattr(config, "vision_lora")  # Vision LoRA configurations
-        or hasattr(config, "audio_processor")  # Audio processing capabilities
-        or (
-            hasattr(config, "embd_layer") and hasattr(config.embd_layer, "image_embd_layer")
-        )  # Image embedding layers
         or getattr(config, "is_encoder_decoder", False)  # Encoder-decoder VL models
         or any(  # Architecture-based detection for custom VL models (e.g., Nemotron-Parse)
             "conditionalgeneration" in arch.lower() for arch in getattr(config, "architectures", [])
@@ -663,6 +657,41 @@ def _resolve_init_config(hf_config, auto_model_module, ckpt_path, config_kwargs)
         return hf_config
 
 
+def _build_meta_skeleton(from_config, config_for_init, model_kwargs, architecture):
+    """Build a throwaway meta-device model used only to size ``infer_auto_device_map``.
+
+    ``compute_module_sizes`` needs shapes and dtypes, never values or storage, so this
+    probe only has to reproduce the module tree. It must also be *no stricter than the
+    loader it predicts*, or it kills runs ``from_pretrained`` would have completed:
+
+    - Transformers 4.x builds under ``init_empty_weights()``, i.e. ``include_buffers=False``.
+    - Transformers 5.x builds under ``torch.device("meta")`` plus
+      ``meta_device_safe_creation_ops()``, which redirects ``torch.linspace`` to CPU so
+      remote code that derives scalars from it in ``__init__`` keeps working.
+
+    ``include_buffers=True`` is stricter than both: accelerate implements it as a bare
+    global ``torch.device("meta")`` context, so *every* tensor built in ``__init__`` lands
+    on meta and any ``int(torch.tensor(...))`` raises. ``include_buffers=False`` patches
+    only ``nn.Module.register_parameter``, leaving ``__init__`` arithmetic on a real
+    device; buffers stay materialized, which costs nothing here because their shape and
+    dtype size the same either way.
+
+    Returns ``None`` (after warning) if the model cannot be built at all; the caller then
+    skips the estimate rather than failing a load ``from_pretrained`` can still do.
+    """
+    try:
+        with init_empty_weights(include_buffers=False):
+            return from_config(config_for_init, **model_kwargs)
+    except Exception as e:
+        warnings.warn(
+            f"Could not build a meta-device skeleton of {architecture} ({e!r}). "
+            "Skipping the device-map memory estimate and letting from_pretrained map the "
+            "model. If you hit GPU OOM, rerun with --use_seq_device_map (which applies "
+            "--gpu_max_mem_percentage) or lower --batch_size."
+        )
+        return None
+
+
 def _get_config_dtype(config):
     config_dtype = (
         getattr(config, "dtype", None) or getattr(config, "torch_dtype", None) or torch.bfloat16
@@ -873,17 +902,19 @@ def has_pack_quantized_config(config):
             hf_config, auto_model_module, ckpt_path, config_kwargs
         )
 
-        with init_empty_weights(include_buffers=True):
-            # When computing the device_map, assuming bfloat16 precision by default,
-            # unless specified by the hf_config.
-            config_dtype = _get_config_dtype(config_for_init)
-            model_kwargs2 = _apply_dtype_to_config(
-                model_kwargs, config_dtype, architecture, apply_config_dtype=True
-            )
-            if auto_model_module not in [AutoModelForCausalLM, AutoModel]:
-                model_kwargs2.pop("trust_remote_code", None)
-            model_kwargs2.pop("max_memory", None)
-            model = from_config(config_for_init, **model_kwargs2)
+        # When computing the device_map, assuming bfloat16 precision by default,
+        # unless specified by the hf_config.
+        config_dtype = _get_config_dtype(config_for_init)
+        model_kwargs2 = _apply_dtype_to_config(
+            model_kwargs, config_dtype, architecture, apply_config_dtype=True
+        )
+        if auto_model_module not in [AutoModelForCausalLM, AutoModel]:
+            model_kwargs2.pop("trust_remote_code", None)
+        model_kwargs2.pop("max_memory", None)
+
+        # Only a sizing aid for ``infer_auto_device_map`` below; ``None`` when the model
+        # cannot be built on meta, in which case the estimate is skipped.
+        model = _build_meta_skeleton(from_config, config_for_init, model_kwargs2, architecture)
 
         max_memory = get_max_memory()
 
@@ -903,7 +934,7 @@ def has_pack_quantized_config(config):
                 f"Offload folder: {offload_folder}\n"
                 "Weights exceeding GPU+CPU budgets will be streamed from disk."
             )
-        else:
+        elif model is not None:
             inferred_device_map = infer_auto_device_map(model, max_memory=max_memory)
             if "cpu" in inferred_device_map.values():
                 for _device in max_memory:

examples/hf_ptq/hf_ptq.py

@@ -705,9 +705,6 @@ def load_model(args: argparse.Namespace):
         # Left padding usually provides better calibration result.
         tokenizer.padding_side = "left"
 
-    if model_type == "phi4mm":
-        warnings.warn("Please set the default input_mode to InputMode.LANGUAGE before quantizing.")
-
     return (
         full_model,
         language_model,

modelopt/torch/export/layer_utils.py

@@ -222,12 +222,7 @@ def is_conv(module: nn.Module) -> bool:
 def is_embedding(module: nn.Module) -> bool:
     """Returns whether the module is an embedding layer."""
     module_type_name = type(module).__name__
-    return (
-        "Embedding" in module_type_name
-        and "Rotary" not in module_type_name
-        and "PhiImage" not in module_type_name
-        and "Phi3Image" not in module_type_name
-    )
+    return "Embedding" in module_type_name and "Rotary" not in module_type_name
 
 
 def build_embedding_config(module: nn.Module, normalization_constant: float = 1) -> EmbeddingConfig:

modelopt/torch/export/model_utils.py

@@ -44,7 +44,6 @@
     "phi3small": "phi3small",
     "phi3": "phi3",
     "PhiMoEForCausalLM": "phi3",
-    "Phi4MMForCausalLM": "phi4mm",
     "phi": "phi",
     "TLGv4ForCausalLM": "phi",
     "MixtralForCausalLM": "llama",
@@ -88,10 +87,6 @@ def is_multimodal_model(model):
     This function detects various multimodal model architectures by checking for:
     - Standard vision configurations (vision_config)
     - Language model attributes (language_model)
-    - Specific multimodal model types (phi4mm)
-    - Vision LoRA configurations
-    - Audio processing capabilities
-    - Image embedding layers
     - Nemotron-Parse conditional generation models
 
     Args:
@@ -104,10 +99,6 @@ def is_multimodal_model(model):
         >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
         >>> is_multimodal_model(model)
         True
-
-        >>> model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-4-multimodal-instruct")
-        >>> is_multimodal_model(model)
-        True
     """
     config = model.config
 
@@ -118,12 +109,6 @@ def is_multimodal_model(model):
     return (
         hasattr(config, "vision_config")  # Standard vision config (e.g., Qwen2.5-VL)
         or hasattr(model, "language_model")  # Language model attribute (e.g., LLaVA)
-        or getattr(config, "model_type", "") == "phi4mm"  # Phi-4 multimodal
-        or hasattr(config, "vision_lora")  # Vision LoRA configurations
-        or hasattr(config, "audio_processor")  # Audio processing capabilities
-        or (
-            hasattr(config, "embd_layer") and hasattr(config.embd_layer, "image_embd_layer")
-        )  # Image embedding layers
         or is_nemotron_parse  # Nemotron-Parse conditional generation model
     )