Add FP8 Vision Encoder quantization for Qwen3-VL and Qwen3.5 - #2083
Add FP8 Vision Encoder quantization for Qwen3-VL and Qwen3.5#2083mpariente-nvda wants to merge 1 commit into
Conversation
…and Qwen3.5 Signed-off-by: mpariente <mpariente@nvidia.com>
📝 WalkthroughWalkthroughChangesQwen Vision FP8 PTQ
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant hf_ptq.py
participant ImageCalibrationDataloader
participant FullVLM
participant Quantizer
hf_ptq.py->>ImageCalibrationDataloader: load image-text batches
ImageCalibrationDataloader->>FullVLM: forward multimodal batches
FullVLM->>Quantizer: instrument selected quantization target
Quantizer-->>hf_ptq.py: return quantized model
🚥 Pre-merge checks | ✅ 5 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warning
CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.
Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/hf_ptq/hf_ptq.py`:
- Around line 666-674: Require --calib_with_images when running vision FP8
recipes, including the corresponding validation path near the
recipe/auto-quantization handling and the related logic around this block.
Reject the configuration with a clear error before calibration starts, while
preserving existing behavior for non-vision recipes and image-enabled runs, and
add a regression test covering the rejection.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 5ed83f54-53df-4161-bb6a-ff16600c34dd
📒 Files selected for processing (13)
CHANGELOG.rstexamples/hf_ptq/README.mdexamples/hf_ptq/hf_ptq.pymodelopt_recipes/huggingface/qwen3_5/ptq/fp8_vision-kv_none.yamlmodelopt_recipes/huggingface/qwen3_5/ptq/fp8_vision_lm-kv_fp8_cast.yamlmodelopt_recipes/huggingface/qwen3_vl/ptq/fp8_vision-kv_none.yamlmodelopt_recipes/huggingface/qwen3_vl/ptq/fp8_vision_lm-kv_fp8_cast.yamlmodelopt_recipes/huggingface/qwen3_vl/ptq/vision_fp8.quant_cfg.yamlmodelopt_recipes/ptq.mdtests/_test_utils/torch/transformers_models.pytests/examples/hf_ptq/test_hf_ptq_vision_quantization.pytests/gpu/torch/export/test_qwen_vision_recipe_export.pytests/unit/recipe/test_qwen_vision_recipe.py
| # Plain PTQ quantizes only the language model. Recipes and AutoQuantize keep the complete | ||
| # VLM so their quantizer rules can target vision and language components in one state. | ||
| if args.recipe is None and args.auto_quantize_bits is None: | ||
| extracted_lm, extracted_model_type = extract_and_prepare_language_model_from_vl( | ||
| full_model | ||
| ) | ||
| if extracted_lm is not None: | ||
| language_model = extracted_lm | ||
| model_type = extracted_model_type |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Require image calibration for the vision FP8 recipes.
Without --calib_with_images, this flow retains the complete VLM but creates a text-only calibration loop. That loop does not execute visual, so the enabled *visual.*input_quantizer entries receive no activation calibration data. Reject vision FP8 recipes without --calib_with_images, and add a regression test for that error.
Also applies to: 788-792
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/hf_ptq/hf_ptq.py` around lines 666 - 674, Require
--calib_with_images when running vision FP8 recipes, including the corresponding
validation path near the recipe/auto-quantization handling and the related logic
around this block. Reject the configuration with a clear error before
calibration starts, while preserving existing behavior for non-vision recipes
and image-enabled runs, and add a regression test covering the rejection.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2083 +/- ##
==========================================
+ Coverage 67.15% 67.16% +0.01%
==========================================
Files 521 521
Lines 59857 59857
==========================================
+ Hits 40199 40205 +6
+ Misses 19658 19652 -6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| # Image calibration batches contain multimodal kwargs (for example pixel_values). | ||
| # They must be consumed by the complete VLM even when only a nested component is the | ||
| # quantization target; the full forward still exercises that component's quantizers. | ||
| if args.calib_with_images: |
There was a problem hiding this comment.
Thanks @Edwardf0t1 , overall looks good to me. I have a comment though about the behavior change for non-Nemotron VLM models that were previously running with -calib-with-image flag
Before this PR, users were (unknowingly) getting a text-only calibration loop even with the flag set. After this PR, the full VLM forward runs — pixel values go through the vision encoder, get projected into the LM's embedding space, and the LM quantizers see those activations. That changes the amax distributions and therefore the output scales, even when the vision encoder itself has no quantizers enabled.
So a Qwen2.5-VL or Phi-4-multimodal user running the same --calib_with_images PTQ command before and after this PR will get a different quantized checkpoint.
If my understanding is correct, should we mention this somewhere in the CHANGELOG so the users are aware of this behavior change?
There was a problem hiding this comment.
Yes, the full VLM forward now makes the LM quantizers observe vision conditioned activations, so existing --calib_with_images commands may produce different activation ranges and scales. The vision branch still remains unquantized unless explicitly enabled by a recipe. The calibration fix is briefly mentioned in the CHANGELOG, but I can expand the entry to make the checkpoint impact explicit.
2 participants
What does this PR do?
Type of change: New feature
Adds opt-in FP8 Vision Encoder quantization recipes for Qwen3-VL and dense Qwen3.5:
fp8_vision-kv_none: FP8 Vision Encoder Linears, with the LLM and KV cache kept in high precision.fp8_vision_lm-kv_fp8_cast: FP8 Vision Encoder and LLM Linears, with FP8 KV-cache cast.Patch embedding and vision-attention BMM operands remain in high precision. Recipe-based image calibration now keeps the complete VLM so multimodal inputs exercise the selected vision and language quantizers. The existing non-recipe VLM path remains language-model-only.
Usage
For dense Qwen3.5, replace
qwen3_vlwithqwen3_5in the recipe path.Testing
git diff --checkpassed.Deployment evidence with Qwen3-VL-2B on RTX PRO 6000 BSE, eight fixed frames and a BF16 LLM:
The accuracy mean covers MMMU, RealWorldQA, Video-MMMU, MVBench, and Video-MME. Serving reached 7.7% lower end-to-end latency and 7.9% higher throughput at concurrency 16.
Runtime support for quantized Vision Encoder Linears is separate from this ModelOpt checkpoint-generation change.
Before your PR is "Ready for review"
CONTRIBUTING.md: N/A/claude reviewafter opening the PR.Additional Information
Summary by CodeRabbit