Skip to content

docs: rebuild the unified HF deployment support matrix from the deploy test suite (NVBug 6550792) #2087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Edwardf0t1 wants to merge 2 commits into
base: main
Choose a base branch
from
+161 −85
Open

docs: rebuild the unified HF deployment support matrix from the deploy test suite (NVBug 6550792) #2087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3a47fc1
docs: rebuild the unified HF deployment support matrix from the deplo…
Edwardf0t1 Aug 5, 2026
5742dba
docs: address review — TRT-LLM v1.2.0 floor, checkmarks, conditional …
Edwardf0t1 Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 12 additions & 23 deletions .agents/skills/deployment/references/support-matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,14 @@

## Unified HF Checkpoint — Framework Compatibility

| Model | Quant Format | TRT-LLM | vLLM | SGLang |
|-------|-------------|---------|------|--------|
| Llama 3.x | FP8 | yes | yes | yes |
| Llama 3.x | FP4 | yes | yes | yes |
| Llama 4 | FP8 | yes | — | yes |
| Llama 4 | FP4 | yes | — | — |
| DeepSeek R1 | FP8 | yes | yes | yes |
| DeepSeek R1 | FP4 | yes | yes | yes |
| DeepSeek V3 | FP8 | yes | yes | yes |
| DeepSeek V3 | FP4 | yes | yes | yes |
| Qwen 3 | FP8 | yes | yes | yes |
| Qwen 3 | FP4 | yes | yes | — |
| Qwen 3 MoE | FP8 | yes | yes | yes |
| Qwen 3 MoE | FP4 | yes | — | — |
| Qwen 2.5 | FP8 | yes | yes | yes |
| Qwen 2.5 | FP4 | yes | yes | — |
| QwQ-32B | FP8 | yes | yes | yes |
| QwQ-32B | FP4 | yes | yes | — |
| Mixtral 8x7B | FP8 | yes | yes | yes |
| Mixtral 8x7B | FP4 | yes | — | — |
**Do not maintain a copy of the matrix here.** The single source of truth is
`docs/source/deployment/3_unified_hf.rst` ("Model Support Matrix"), and every validated entry in it
is backed by `tests/examples/hf_ptq/test_deploy.py`.

To answer "is model X supported on framework Y", read one of those two files — `test_deploy.py` is
the more precise answer, since it also carries the exact checkpoint, tensor-parallel size, and
minimum SM version per entry. It covers language models, VLMs (Qwen2.5-VL, Qwen3-VL,
Phi-4-multimodal, Nemotron Omni), EAGLE3/Medusa drafters, and diffusion models.

## Supported Quantization Formats

Expand Down Expand Up @@ -50,9 +38,9 @@
| SGLang | `quantization="modelopt"` | `quantization="modelopt_fp4"` |
| TRT-LLM | auto-detected from checkpoint | auto-detected from checkpoint |

## Models not in this list
## Models not in the matrix

This matrix covers officially validated combinations. For unlisted models:
The matrix covers officially validated combinations, not the full set of what will run. For unlisted models:

1. **Check the framework's own docs** — vLLM and SGLang support many HuggingFace models natively. Use WebSearch to check `vllm supported models` or `sglang supported models`.
2. **Try it** — if the model uses standard `nn.Linear` layers and has `hf_quant_config.json`, vLLM/SGLang will likely work with `--quantization modelopt`.
Expand All @@ -64,4 +52,5 @@ This matrix covers officially validated combinations. For unlisted models:
- **B300/GB300 are `sm_103`** and need a **CUDA-13** serving image — from v0.20.0 the unsuffixed tag is CUDA-13 (`-cu129` opts back to CUDA 12); `cu12` images lack the `sm_103` FP4 kernel and serve NVFP4 as gibberish or error out. See the CUDA-13 note in the deployment `SKILL.md`.
- **Verify the GPU with `nvidia-smi`** before choosing the image — cluster GPU labels can be stale.
- INT4_AWQ and W4A8_AWQ are only supported by TRT-LLM (not vLLM or SGLang).
- Source: `examples/hf_ptq/README.md` and `docs/source/deployment/3_unified_hf.rst`
- For VLMs, only the language model is quantized; the vision encoder stays in high precision, so multimodal serving depends on the framework's own support for that architecture.
- Source: `docs/source/deployment/3_unified_hf.rst` and `tests/examples/hf_ptq/test_deploy.py`
172 changes: 133 additions & 39 deletions docs/source/deployment/3_unified_hf.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,48 +51,140 @@ The unified HF export API supports the following quantization formats:
5. INT4_AWQ - 4-bit integer with AWQ optimization
6. W4A8_AWQ - 4-bit weights and 8-bit activations with AWQ optimization

Framework-Specific Support
Minimum Framework Versions
--------------------------

TensorRT-LLM
~~~~~~~~~~~~
=============== =================
Framework Minimum version
=============== =================
TensorRT-LLM v1.2.0
vLLM v0.10.1
SGLang v0.4.10
=============== =================

These are the oldest versions expected to load a unified HF checkpoint. The matrix below is
validated against the versions CI currently runs, which are newer — see the container tags in
``.github/workflows/``.

.. _unified-hf-support-matrix:

Model Support Matrix
--------------------

Legend:

* ✅ — covered by the release deployment test suite
(`tests/examples/hf_ptq/test_deploy.py <https://github.com/NVIDIA/Model-Optimizer/blob/main/tests/examples/hf_ptq/test_deploy.py>`_),
which loads the exported checkpoint in the framework and runs generation.
* ⚠ — documented as working previously but not in the current test suite; expected to work, unvalidated.
* ``-`` — not currently covered. It may still work; see `Models not listed here`_.

Language models
~~~~~~~~~~~~~~~

============================================ ============== ============ ====== ========
Model Quant format TensorRT-LLM vLLM SGLang
============================================ ============== ============ ====== ========
Llama 3.1, 3.3 FP8, NVFP4 ✅ ✅ ✅
Llama 4 Scout, Maverick FP8 ✅ ✅ ✅
Llama 4 Scout NVFP4 ✅ ✅ ✅
Llama Nemotron Super 49B v1, v1.5 FP8 ✅ ✅ ✅
Llama Nemotron Ultra 253B v1 FP8 ✅ ✅ ✅
Nemotron 3 Nano 30B-A3B FP8, NVFP4 ✅ ✅ ✅
Nemotron 3 Super 120B-A12B FP8, NVFP4 ✅ ✅ ✅
Nemotron 3 Ultra 550B-A55B NVFP4 ✅ ✅ ✅
DeepSeek R1, R1-0528 NVFP4 ✅ ✅ ✅
DeepSeek V3, V3.1, V3.2 NVFP4 ✅ ✅ ✅
DeepSeek V4 Flash NVFP4 ✅ ✅ ✅
DeepSeek V4 Pro NVFP4 \- ✅ ✅
Qwen 3 (8B, 14B, 32B) FP8, NVFP4 ✅ ✅ ✅
Qwen 3 MoE 235B-A22B FP8, NVFP4 ✅ ✅ ✅
Qwen 3 MoE 30B-A3B NVFP4 ✅ ✅ ✅
Qwen 3 Coder 480B-A35B NVFP4 ✅ ✅ ✅
Qwen 3-Next 80B-A3B NVFP4 ✅ ✅ ✅
Qwen 3.5 397B-A17B NVFP4 ✅ ✅ ✅
Qwen 3.5 122B-A10B, Qwen 3.6 35B-A3B NVFP4 \- ✅ \-
Qwen 2.5 FP8 ⚠ ⚠ ⚠
Qwen 2.5 NVFP4 ⚠ ⚠ \-
QwQ-32B FP8 ⚠ ⚠ ⚠
QwQ-32B NVFP4 ⚠ ⚠ \-
Phi-4 reasoning-plus FP8, NVFP4 ✅ ✅ ✅
Gemma 4 31B NVFP4 ✅ ✅ ✅
Gemma 4 26B-A4B NVFP4 \- ✅ \-
GLM-4.7, GLM-5, GLM-5.2 NVFP4 ✅ ✅ ✅
GLM-5.1 NVFP4 \- ✅ ✅
Kimi K2-Thinking, K2.5 NVFP4 ✅ ✅ ✅
Kimi K2.6 NVFP4 \- ✅ \-
MiniMax M2.5, M3 NVFP4 ✅ ✅ ✅
Mixtral 8x7B FP8 ⚠ ⚠ ⚠
Mixtral 8x7B NVFP4 ⚠ \- \-
============================================ ============== ============ ====== ========

Vision-language and multimodal models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For VLMs, modelopt quantizes the language model only; the vision encoder is kept in high precision.
The exported checkpoint therefore relies on the serving framework's own multimodal support for that
architecture — see the
`TensorRT-LLM multimodal support matrix <https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/models/supported-models.md#multimodal-feature-support-matrix-pytorch-backend>`_.

============================================ ============== ============ ====== ========
Model Quant format TensorRT-LLM vLLM SGLang
============================================ ============== ============ ====== ========
Qwen 2.5-VL 7B FP8, NVFP4 ✅ ✅ ✅
Qwen 3-VL 235B-A22B NVFP4 ✅ ✅ ✅
Phi-4-multimodal FP8, NVFP4 ✅ ✅ ✅
Nemotron 3 Nano Omni 30B-A3B FP8, NVFP4 ✅ ✅ ✅
============================================ ============== ============ ====== ========

Speculative decoding drafters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Drafters are deployed on top of their base checkpoint. vLLM is not currently covered for these.

Unlike the tables above, drafter coverage is conditional: the EAGLE3 tests run only when
``MODELOPT_LOCAL_EAGLE_MODEL`` points at a directory containing the drafter, and skip otherwise.
✅ below means "covered when that drafter is available locally".

============================================================ ============ ============ ====== ========
Drafter Quant format TensorRT-LLM vLLM SGLang
============================================================ ============ ============ ====== ========
EAGLE3 for Llama 3.3 70B, Llama 4 Maverick FP8 ✅ \- ✅
EAGLE3 for Qwen 3 235B-A22B (incl. Thinking-2507, FP4) BF16, NVFP4 ✅ \- ✅
EAGLE3 for Qwen 3 30B-A3B-Thinking-2507 BF16 ✅ \- ✅
EAGLE3 for Kimi K2-Thinking, K2.5, K2.6 NVFP4 ✅ \- ✅
EAGLE3 for gpt-oss-120b BF16 ✅ \- ✅
Medusa for Llama 3.1 8B FP8 ✅ \- ✅
============================================================ ============ ============ ====== ========

Diffusion models
~~~~~~~~~~~~~~~~

============================================ ============== ============ ====== ========
Model Quant format TensorRT-LLM vLLM SGLang
============================================ ============== ============ ====== ========
Wan 2.2 T2V A14B FP8, NVFP4 ✅ \- ✅
DiffusionGemma 26B-A4B NVFP4 ✅ ✅ ✅
============================================ ============== ============ ====== ========

Models:
* Llama 4, 3.x (FP8, NVFP4)
* Qwen 3, 2.5 (FP8, NVFP4)
* Qwen 3 MoE (FP8, NVFP4)
* Qwen 3-VL (FP8, NVFP4)
* Deepseek R1/V3 (NVFP4)
* Mixtral 8x7B (FP8, NVFP4)
* Medusa (FP8)
* Eagle (FP8)

Requirements: TensorRT-LLM v0.17.0 or later

vLLM
~~~~

Models:
* Llama 4, 3.x (FP8, NVFP4)
* Qwen 3, 2.5 (FP8, NVFP4)
* Qwen 3 MoE (FP8, NVFP4)
* Mixtral 8x7B (FP8)
* Deepseek R1/V3 (NVFP4)

Requirements: vLLM v0.10.1 or later

SGLang
~~~~~~
.. note::
NVFP4 inference requires Blackwell GPUs. Hopper can produce an NVFP4 checkpoint but cannot serve
it. On B300/GB300 (``sm_103``) use a CUDA-13 build of the serving framework; CUDA-12 builds lack
the ``sm_103`` FP4 kernels.

Models:
* Llama 4, 3.x (FP8, NVFP4)
* Qwen 3, 2.5 (FP8, NVFP4)
* Qwen 3 MoE (FP8, NVFP4)
* Deepseek R1/V3 (NVFP4)
Models not listed here
~~~~~~~~~~~~~~~~~~~~~~

Requirements: SGLang v0.4.10 or later
This matrix records the combinations modelopt validates. It is not an exhaustive list of what will
run: vLLM, SGLang, and TensorRT-LLM load unified HF checkpoints generically, so a model built from
standard ``nn.Linear`` layers with an ``hf_quant_config.json`` will often deploy without any modelopt
change. Check the serving framework's own model support list first, then try it.

Note: While other models and quantization formats may work, they have not been thoroughly tested and validated.
The exact checkpoints behind every ``Y`` above, including tensor-parallel size and minimum SM
version, are listed in
`tests/examples/hf_ptq/test_deploy.py <https://github.com/NVIDIA/Model-Optimizer/blob/main/tests/examples/hf_ptq/test_deploy.py>`__;
most are published under the
`NVIDIA Hugging Face organization <https://huggingface.co/nvidia>`_.


Deployment with Selected Inference Frameworks
Expand All @@ -102,7 +194,7 @@ Deployment with Selected Inference Frameworks

Follow the `TensorRT-LLM installation instructions. <https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html#installation>`_

Currently we support fp8 and nvfp4 quantized models for TensorRT-LLM deployment, you need v0.17.0 or later version of TensorRT-LLM.
FP8 and NVFP4 quantized models are supported; you need v1.2.0 or later version of TensorRT-LLM.

To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below:

Expand Down Expand Up @@ -136,7 +228,8 @@ Deployment with Selected Inference Frameworks

Follow `vLLM installation instructions. <https://github.com/vllm-project/vllm?tab=readme-ov-file#getting-started>`_

Currently we support fp8 quantized models (without fp8 kv cache) for vLLM deployment, you need v0.6.5 or later version of vLLM.
FP8 and NVFP4 quantized models are supported; you need v0.10.1 or later version of vLLM. Pass
``quantization="modelopt"`` for FP8 and ``quantization="modelopt_fp4"`` for NVFP4.

To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below:

Expand Down Expand Up @@ -171,7 +264,8 @@ Deployment with Selected Inference Frameworks

Follow the `SGLang installation instructions. <https://docs.sglang.ai/get_started/install.html>`_

Currently we support fp8 quantized models (without fp8 kv cache) for SGLang deployment, you need to use the main branch of SGLang (since Jan 6, 2025) and build it from source.
FP8 and NVFP4 quantized models are supported; you need v0.4.10 or later version of SGLang. Pass
``quantization="modelopt"`` for FP8 and ``quantization="modelopt_fp4"`` for NVFP4.

To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below:

Expand Down
2 changes: 1 addition & 1 deletion examples/diffusers/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ mtq.quantize(model=transformer, config=quant_config, forward_func=forward_pass)

> *<sup>1.</sup>The w4a8_awq is an experimental quantization scheme that may result in a higher accuracy penalty.*

> *<sup>2.</sup>A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v0.17 or later*
> *<sup>2.</sup>A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later*

> *<sup>3.</sup>The SVDQuant Perf in TRT might not good as the [Nunchaku: MIT-Nvidia](https://github.com/nunchaku-tech/nunchaku) at this moment.*

Expand Down
37 changes: 15 additions & 22 deletions examples/hf_ptq/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ Please reference our [framework scripts](#framework-scripts) and our [docs](http
> *<sup>2.</sup>For some models, there is only support for exporting quantized checkpoints.* \
> *<sup>3.</sup>W4A8_AWQ is only available on some models but not all* \
> *<sup>4.</sup>For some models, KV cache quantization may result in a higher accuracy penalty.* \
> *<sup>5.</sup>A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v0.17 or later* \
> *<sup>5.</sup>A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later* \
> *<sup>6.</sup>Some models currently support export to HF format only.* \
> *<sup>7.</sup>[PTQ for DeepSeek](../deepseek/README.md)* \
> *<sup>8.</sup>GLM-4.7 has MTP (Multi-Token Prediction) layers that are automatically loaded and excluded from quantization.* \
Expand Down Expand Up @@ -588,27 +588,20 @@ print(llm_fp8.generate(["What's the age of the earth? "]))

### Unified HF Checkpoint Deployment Model Support Matrix

| Model | Quant format | TRT-LLM | vLLM | SGLang |
| :---: | :---: | :---: | :---: | :---: |
| LLAMA 3.x | FP8 | ✅ | ✅ | ✅ |
| LLAMA 3.x | FP4 | ✅ | ✅ | ✅ |
| LLAMA 4 | FP8 | ✅ | - | ✅ |
| LLAMA 4 | FP4 | ✅ | - | - |
| DS-R1 | FP8 | ✅ | ✅ | ✅ |
| DS-R1 | FP4 | ✅ | ✅ | ✅ |
| DS-V3 | FP8 | ✅ | ✅ | ✅ |
| DS-V3 | FP4 | ✅ | ✅ | ✅ |
| QWen3 | FP8 | ✅ | ✅ | ✅ |
| QWen3 | FP4 | ✅ | ✅ | - |
| QWen3 MoE | FP8 | ✅ | ✅ | ✅ |
| QWen3 MoE | FP4 | ✅ | - | - |
| QWen3.5 MoE | FP4 | - | - | ✅ |
| QWen2.5 | FP8 | ✅ | ✅ | ✅ |
| QWen2.5 | FP4 | ✅ | ✅ | - |
| QwQ-32B | FP8 | ✅ | ✅ | ✅ |
| QwQ-32B | FP4 | ✅ | ✅ | - |
| Mixtral 8x7B | FP8 | ✅ | ✅ | ✅ |
| Mixtral 8x7B | FP4 | ✅ | - | - |
The deployment support matrix — which model families and quantization formats are validated on
TRT-LLM, vLLM, and SGLang, including vision-language models, speculative decoding drafters, and
diffusion models — lives in the documentation so there is a single copy to keep current:

**[Unified HF Checkpoint → Model Support Matrix](https://nvidia.github.io/Model-Optimizer/deployment/3_unified_hf.html#model-support-matrix)**

Every validated entry there is backed by [`tests/examples/hf_ptq/test_deploy.py`](../../tests/examples/hf_ptq/test_deploy.py),
which loads the exported checkpoint in each framework and runs generation. That file is also the
place to look for the exact checkpoint, tensor-parallel size, and minimum SM version behind each
entry.

> *Note: the matrix records what modelopt validates, not the full set of what will run. vLLM, SGLang,
> and TRT-LLM load unified HF checkpoints generically, so unlisted models frequently deploy without
> any modelopt change — check the serving framework's own model support list and try it.*

### (Legacy) TensorRT-LLM Checkpoints

Expand Down
Loading