Nemotron3: Failed to load model with all configured dtypes.

[mcr-ksh]

When trying Nemotron nvfp4 quantizised version.
Same issue with nvfp4 or auto.

Any hint how todo it?
Got the model running on vLLM v0.14.1
transformers==4.57.6
numpy==2.2

# heretic --dtypes nvfp4 --trust-remote-code true --model cybermotaz/nemotron3-nano-nvfp4-w4a16 
█░█░█▀▀░█▀▄░█▀▀░▀█▀░█░█▀▀  v1.1.0
█▀█░█▀▀░█▀▄░█▀▀░░█░░█░█░░
▀░▀░▀▀▀░▀░▀░▀▀▀░░▀░░▀░▀▀▀  https://github.com/p-e-w/heretic

Detected 1 CUDA device(s):
* GPU 0: NVIDIA Thor

Loading model cybermotaz/nemotron3-nano-nvfp4-w4a16...
* Trying dtype nvfp4... Failed (NemotronHForCausalLM.__init__() got an unexpected keyword argument 'dtype')
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/local/miniconda3/envs/vllm/bin/heretic:10 in <module>                                       │
│                                                                                                  │
│    7 │   │   sys.argv[0] = sys.argv[0][:-11]                                                     │
│    8 │   elif sys.argv[0].endswith(".exe"):                                                      │
│    9 │   │   sys.argv[0] = sys.argv[0][:-4]                                                      │
│ ❱ 10 │   sys.exit(main())                                                                        │
│   11                                                                                             │
│                                                                                                  │
│ /usr/src/heretic/src/heretic/main.py:888 in main                                                 │
│                                                                                                  │
│   885 │   install()                                                                              │
│   886 │                                                                                          │
│   887 │   try:                                                                                   │
│ ❱ 888 │   │   run()                                                                              │
│   889 │   except BaseException as error:                                                         │
│   890 │   │   # Transformers appears to handle KeyboardInterrupt (or BaseException)              │
│   891 │   │   # internally in some places, which can re-raise a different error in the handler   │
│                                                                                                  │
│ /usr/src/heretic/src/heretic/main.py:307 in run                                                  │
│                                                                                                  │
│   304 │   │   elif choice is None or choice == "":                                               │
│   305 │   │   │   return                                                                         │
│   306 │                                                                                          │
│ ❱ 307 │   model = Model(settings)                                                                │
│   308 │   print()                                                                                │
│   309 │   print_memory_usage()                                                                   │
│   310                                                                                            │
│                                                                                                  │
│ /usr/src/heretic/src/heretic/model.py:145 in __init__                                            │
│                                                                                                  │
│   142 │   │   │   break                                                                          │
│   143 │   │                                                                                      │
│   144 │   │   if self.model is None:                                                             │
│ ❱ 145 │   │   │   raise Exception("Failed to load model with all configured dtypes.")            │
│   146 │   │                                                                                      │
│   147 │   │   self._apply_lora()                                                                 │
│   148                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Exception: Failed to load model with all configured dtypes.

[Vinay-Umrethe]

this type of quantization is likely not supported, a PR was made by @accemlcc IG which supported bitsandbytes 4bit quant

[mcr-ksh]

# heretic --trust-remote-code true --model cybermotaz/nemotron3-nano-nvfp4-w4a16 --quantization BNB_4BIT --dtypes float16
█░█░█▀▀░█▀▄░█▀▀░▀█▀░█░█▀▀  v1.1.0
█▀█░█▀▀░█▀▄░█▀▀░░█░░█░█░░
▀░▀░▀▀▀░▀░▀░▀▀▀░░▀░░▀░▀▀▀  https://github.com/p-e-w/heretic

GPU type: NVIDIA Thor

Loading model cybermotaz/nemotron3-nano-nvfp4-w4a16...
Loading checkpoint shards:   0%|                                                                                                                                                                                             | 0/4 [00:00<?, ?it/s]
Failed (Blockwise 4bit quantization only supports 16/32-bit floats, but got torch.uint8)

Also @accemlcc 4bit doesn't work

[Vinay-Umrethe]

@mcr-ksh you tried to quant a model which is already with a different type, change it to this base model with 4 bit bnb

nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

[nonetrix]

I am on AMD so might be just AMD skill issue but I tried it with BF16 and it doesn't work sadly :/

~/Documents/AI/heretic 14s
venv ❯ heretic nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
█░█░█▀▀░█▀▄░█▀▀░▀█▀░█░█▀▀  v1.1.0
█▀█░█▀▀░█▀▄░█▀▀░░█░░█░█░░
▀░▀░▀▀▀░▀░▀░▀▀▀░░▀░░▀░▀▀▀  https://github.com/p-e-w/heretic

/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
GPU type: AMD Radeon Graphics

Loading model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16...
config.json: 1.82kB [00:00, 7.33MB/s]
The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
configuration_nemotron_h.py: 12.9kB [00:00, 53.4MB/s]
tokenizer_config.json: 188kB [00:00, 691MB/s]
tokenizer.json: 100%|██████████████████████████████████████████████| 17.1M/17.1M [00:00<00:00, 19.9MB/s]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
special_tokens_map.json: 100%|█████████████████████████████████████████| 420/420 [00:00<00:00, 3.02MB/s]
chat_template.jinja: 10.5kB [00:00, 45.8MB/s]
* Trying dtype auto... The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
modeling_nemotron_h.py: 83.6kB [00:00, 259MB/s]
Failed (mamba-ssm is required by the Mamba model but cannot be imported)
* Trying dtype float16... The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
Failed (mamba-ssm is required by the Mamba model but cannot be imported)
* Trying dtype bfloat16... The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
Failed (mamba-ssm is required by the Mamba model but cannot be imported)
* Trying dtype float32... The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
The repository nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 .
 You can inspect the repository content at https://hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
Failed (mamba-ssm is required by the Mamba model but cannot be imported)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/noah/Documents/AI/heretic/venv/bin/heretic:7 in <module>                                   │
│                                                                                                  │
│   4 if __name__ == '__main__':                                                                   │
│   5 │   if sys.argv[0].endswith('.exe'):                                                         │
│   6 │   │   sys.argv[0] = sys.argv[0][:-4]                                                       │
│ ❱ 7 │   sys.exit(main())                                                                         │
│   8                                                                                              │
│                                                                                                  │
│ /home/noah/Documents/AI/heretic/venv/lib/python3.14/site-packages/heretic/main.py:576 in main    │
│                                                                                                  │
│   573 │   install()                                                                              │
│   574 │                                                                                          │
│   575 │   try:                                                                                   │
│ ❱ 576 │   │   run()                                                                              │
│   577 │   except BaseException as error:                                                         │
│   578 │   │   # Transformers appears to handle KeyboardInterrupt (or BaseException)              │
│   579 │   │   # internally in some places, which can re-raise a different error in the handler   │
│                                                                                                  │
│ /home/noah/Documents/AI/heretic/venv/lib/python3.14/site-packages/heretic/main.py:133 in run     │
│                                                                                                  │
│   130 │   # Silence the warning about multivariate TPE being experimental.                       │
│   131 │   warnings.filterwarnings("ignore", category=ExperimentalWarning)                        │
│   132 │                                                                                          │
│ ❱ 133 │   model = Model(settings)                                                                │
│   134 │                                                                                          │
│   135 │   print()                                                                                │
│   136 │   print(f"Loading good prompts from [bold]{settings.good_prompts.dataset}[/]...")        │
│                                                                                                  │
│ /home/noah/Documents/AI/heretic/venv/lib/python3.14/site-packages/heretic/model.py:92 in         │
│ __init__                                                                                         │
│                                                                                                  │
│    89 │   │   │   break                                                                          │
│    90 │   │                                                                                      │
│    91 │   │   if self.model is None:                                                             │
│ ❱  92 │   │   │   raise Exception("Failed to load model with all configured dtypes.")            │
│    93 │   │                                                                                      │
│    94 │   │   print(f"* Transformer model with [bold]{len(self.get_layers())}[/] layers")        │
│    95 │   │   print("* Abliterable components:")                                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Exception: Failed to load model with all configured dtypes.

Off topic but the irony of running a NVIDIA made model on a AMD GPU is not lost on me

[mcr-ksh]

@Vinayyyy7 yes i know, I wanted to use that model as it is optimized for Blackwell and I wouldnt be be able to optimize the model in the same way as the author did. Thus this is requirement for me.

[red40maxxer]

Nemotron3 is hybrid mamba-transformer which isn't supported by heretic, there's a PR for this but it hasn't been updated for a while

[Vinay-Umrethe]

Failed (mamba-ssm is required by the Mamba model but cannot be imported)

yes it's a mamba-ssm model for which you might require mamba-ssm package, although heretic does not support it you can try installing it, or use pure pytorch complied mamba-ssm code

[Vinay-Umrethe]

@Vinayyyy7 yes i know, I wanted to use that model as it is optimized for Blackwell and I wouldnt be be able to optimize the model in the same way as the author did. Thus this is requirement for me.

mamba-models are already so efficient and fast, but since i's hybird of mamba-transformer (they call it jamba) the linear scaling and qudratic scaling mixes up, has the author release the code about their method of quantization, or they kept it private

[cpagac]

@mcr-ksh
I ran into the same issue and built a patch that fixes this, plus several downstream crashes you'll
hit with NemotronH. The core problem, to my understanding, is that "fp8" / "nvfp4" are not valid PyTorch dtypes, so
from_pretrained(dtype="fp8") fails. An idea to fix these is to treat these as special tokens: skip the dtype=
parameter and use torch_dtype=torch.bfloat16 instead, letting HF Transformers auto-detect the
model's built-in quantization_config.

There are also roughly 10 more issues. After fixing the dtype loading, NemotronH will crash on:

1. get_layers() -- NemotronH uses .backbone.layers, not .model.layers
2. get_layer_modules() -- NemotronH has no self_attn or mlp; everything is accessed through .mixer
   with 3 different layer types (Mamba2 SSM, MoE with 128 experts, and Attention)
3. get_abliterable_components() -- only inspects layer 0, which is a Mamba layer, so it misses MoE
   and Attention components entirely
4. get_residuals() -- NemotronH doesn't return hidden_states through generate() (returns a tuple of
   Nones); needs a hook-based fallback
5. Multi-GPU device mismatch -- hooks capture tensors on different CUDA devices, torch.stack() fails
6. Meta-device modules -- if the model doesn't fit entirely in VRAM, offloaded modules produce NaN
   LoRA weights and corrupt the KL divergence score
7. base_refusals == 0 division by zero in evaluator.py

My fork fixes all of these. The changes are in model.py (most of the work), config.py, main.py, and
evaluator.py. (Note: it's sloppy with a bunch of terribly named commits, mainly as I thought I was only coding it for myself)

Usage with the fix:
heretic --dtypes fp8 --trust-remote-code true --model cybermotaz/nemotron3-nano-nvfp4-w4a16

Note: --dtypes fp8 (not nvfp4) -- the fork treats "fp8" as the special token for all FP8/NVFP4
pre-quantized models.

Important VRAM note: This model is ~59GB at bfloat16 (HF loads the NVFP4 weights into bfloat16). On
your Thor (48GB), it will offload ~11GB to CPU/meta device, which causes slow inference (~11
min/trial) and potential NaN issues. For best results, you'd need an 80GB GPU (A100/H100).

Fork: https://github.com/venomx-pentester/heretic
Full changelog: FP8_NVFP4_CHANGES.md in the repo

[mcr-ksh]

@mcr-ksh I ran into the same issue and built a patch that fixes this, plus several downstream crashes you'll hit with NemotronH. The core problem, to my understanding, is that "fp8" / "nvfp4" are not valid PyTorch dtypes, so from_pretrained(dtype="fp8") fails. An idea to fix these is to treat these as special tokens: skip the dtype= parameter and use torch_dtype=torch.bfloat16 instead, letting HF Transformers auto-detect the model's built-in quantization_config.

There are also roughly 10 more issues. After fixing the dtype loading, NemotronH will crash on:

1. get_layers() -- NemotronH uses .backbone.layers, not .model.layers
2. get_layer_modules() -- NemotronH has no self_attn or mlp; everything is accessed through .mixer
   with 3 different layer types (Mamba2 SSM, MoE with 128 experts, and Attention)
3. get_abliterable_components() -- only inspects layer 0, which is a Mamba layer, so it misses MoE
   and Attention components entirely
4. get_residuals() -- NemotronH doesn't return hidden_states through generate() (returns a tuple of
   Nones); needs a hook-based fallback
5. Multi-GPU device mismatch -- hooks capture tensors on different CUDA devices, torch.stack() fails
6. Meta-device modules -- if the model doesn't fit entirely in VRAM, offloaded modules produce NaN
   LoRA weights and corrupt the KL divergence score
7. base_refusals == 0 division by zero in evaluator.py

My fork fixes all of these. The changes are in model.py (most of the work), config.py, main.py, and evaluator.py. (Note: it's sloppy with a bunch of terribly named commits, mainly as I thought I was only coding it for myself)

Usage with the fix: heretic --dtypes fp8 --trust-remote-code true --model cybermotaz/nemotron3-nano-nvfp4-w4a16

Note: --dtypes fp8 (not nvfp4) -- the fork treats "fp8" as the special token for all FP8/NVFP4 pre-quantized models.

Important VRAM note: This model is ~59GB at bfloat16 (HF loads the NVFP4 weights into bfloat16). On your Thor (48GB), it will offload ~11GB to CPU/meta device, which causes slow inference (~11 min/trial) and potential NaN issues. For best results, you'd need an 80GB GPU (A100/H100).

Fork: https://github.com/venomx-pentester/heretic Full changelog: FP8_NVFP4_CHANGES.md in the repo

wow! That sounds way more complicated than just the datatype.
I just managed to give it a try but got stuck again:

(vllm) root@jetson-orin:/devel/heretic-nemotron# heretic --trust-remote-code true --dtypes fp8 cybermotaz/nemotron3-nano-nvfp4-w4a16
█░█░█▀▀░█▀▄░█▀▀░▀█▀░█░█▀▀  v1.2.0
█▀█░█▀▀░█▀▄░█▀▀░░█░░█░█░░
▀░▀░▀▀▀░▀░▀░▀▀▀░░▀░░▀░▀▀▀  https://github.com/p-e-w/heretic

Detected 1 CUDA device(s) (61.37 GB total VRAM):
* GPU 0: Orin (61.37 GB)

Loading model cybermotaz/nemotron3-nano-nvfp4-w4a16...
* Trying dtype fp8... Failed ('str' object has no attribute 'is_floating_point')
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /root/miniconda3/envs/vllm/bin/heretic:10 in <module>                                            │
│                                                                                                  │
│    7 │   │   sys.argv[0] = sys.argv[0][:-11]                                                     │
│    8 │   elif sys.argv[0].endswith(".exe"):                                                      │
│    9 │   │   sys.argv[0] = sys.argv[0][:-4]                                                      │
│ ❱ 10 │   sys.exit(main())                                                                        │
│   11                                                                                             │
│                                                                                                  │
│ /root/miniconda3/envs/vllm/lib/python3.10/site-packages/heretic/main.py:909 in main              │
│                                                                                                  │
│   906 │   install()                                                                              │
│   907 │                                                                                          │
│   908 │   try:                                                                                   │
│ ❱ 909 │   │   run()                                                                              │
│   910 │   except BaseException as error:                                                         │
│   911 │   │   # Transformers appears to handle KeyboardInterrupt (or BaseException)              │
│   912 │   │   # internally in some places, which can re-raise a different error in the handler   │
│                                                                                                  │
│ /root/miniconda3/envs/vllm/lib/python3.10/site-packages/heretic/main.py:318 in run               │
│                                                                                                  │
│   315 │   │   elif choice is None or choice == "":                                               │
│   316 │   │   │   return                                                                         │
│   317 │                                                                                          │
│ ❱ 318 │   model = Model(settings)                                                                │
│   319 │   print()                                                                                │
│   320 │   print_memory_usage()                                                                   │
│   321                                                                                            │
│                                                                                                  │
│ /root/miniconda3/envs/vllm/lib/python3.10/site-packages/heretic/model.py:145 in __init__         │
│                                                                                                  │
│   142 │   │   │   break                                                                          │
│   143 │   │                                                                                      │
│   144 │   │   if self.model is None:                                                             │
│ ❱ 145 │   │   │   raise Exception("Failed to load model with all configured dtypes.")            │
│   146 │   │                                                                                      │
│   147 │   │   self._apply_lora()                                                                 │
│   148                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Exception: Failed to load model with all configured dtypes.

Also without specifying dtypes (auto) I get the same error. This is on a Jetson Orin (64GB).

(vllm) root@jetson-orin:/devel/heretic-nemotron# pip list | grep torch
torch                             2.9.1
torch_memory_saver                0.0.9
torchao                           0.9.0
torchaudio                        2.9.1

(vllm) root@jetson-thor:/devel/heretic-nemotron# heretic --dtypes fp8 --trust-remote-code true --model cybermotaz/nemotron3-nano-nvfp4-w4a16
█░█░█▀▀░█▀▄░█▀▀░▀█▀░█░█▀▀  v1.2.0
█▀█░█▀▀░█▀▄░█▀▀░░█░░█░█░░
▀░▀░▀▀▀░▀░▀░▀▀▀░░▀░░▀░▀▀▀  https://github.com/p-e-w/heretic

Detected 1 CUDA device(s) (122.82 GB total VRAM):
* GPU 0: NVIDIA Thor (122.82 GB)

Loading model cybermotaz/nemotron3-nano-nvfp4-w4a16...
* Trying dtype fp8... Failed ('str' object has no attribute 'is_floating_point')

Same on Jetson Thor.

[cpagac]

@mcr-ksh

I cleaned up some things as per p-e-w's request and added more fail-safes as I had trouble getting consistent results on my end. The original NVIDIA/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 should work, but cybermotaz/nemotron3-nano-nvfp4-w4a16 is a different story — that model is NVFP4/W4A16 quantized, and from what I've seen, it appears to require a version of modelopt transformer_engine to interpret the quantization config at load time. The --dtypes fp8 flag is also problematic since heretic passes that string directly to Hugging Face, which expects a torch.Using a dtype object rather than an arbitrary string. The quantized variant requires a loading path that is out of scope for my PR, though.

If you want to give the branch a try on NVIDIA/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16: venomx-pentester:nemotron-support (on my fork). I couldn't test models of that size because of my 32GB VRAM limit, but it should work logically. If you do test it, please let me know your results!

Side note: NemotronH uses Mamba SSM layers, which require mamba-ssm to run. You'll be prompted automatically with install instructions and any other packages that can speed things up.

The updated PR is #166

[mcr-ksh]

unfortunately the same problem.

(vllm) root@jetson-orin:/tmp# heretic --trust-remote-code true --dtypes fp8 NVIDIA/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
█░█░█▀▀░█▀▄░█▀▀░▀█▀░█░█▀▀  v1.2.0
█▀█░█▀▀░█▀▄░█▀▀░░█░░█░█░░
▀░▀░▀▀▀░▀░▀░▀▀▀░░▀░░▀░▀▀▀  https://github.com/p-e-w/heretic

Detected 1 CUDA device(s) (61.37 GB total VRAM):
* GPU 0: Orin (61.37 GB)

Loading model NVIDIA/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16...
* Trying dtype fp8... Failed ('str' object has no attribute 'is_floating_point')

(vllm) root@jetson-orin:/devel/heretic-nemotron# uv pip list | grep -e mamba -e trans -e hugg
Using Python 3.10.18 environment at: /root/miniconda3/envs/vllm
hf-transfer                       0.1.9
huggingface-hub                   0.36.2
mamba-ssm                         2.3.0
transformers                      4.57.6

(vllm) root@jetson-orin:/devel/heretic-nemotron# git branch
  master
* nemotron-support
(vllm) root@jetson-orin:/devel/heretic-nemotron# git log | head -5
commit ecaf64507eeb2b4b1d6f320fe6accc1533770ec4
Author: cpagac <coleman.pagac@drake.edu>
Date:   Wed Feb 18 00:44:29 2026 -0600

    feat: add NemotronH hybrid model support with multi-GPU VRAM calibration

[cpagac]

unfortunately the same problem.

The issue is the --dtypes fp8 flag. That string gets passed directly to HuggingFace, which expects a torch.dtype object, not an arbitrary string, hence the is_floating_point error. This isn't specific to this branch; it would fail the same way on v1.2.0.

Since the model is BF16 (referenced in the name of the model), just drop that flag entirely -- try:
heretic --trust-remote-code true NVIDIA/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

[mcr-ksh]

I was just editing my last post to reflect on that hence the original post stated about using "fp8" as a dtype.

Just a pitty that cybermotaz/nemotron3-nano-nvfp4-w4a16 is not working. That model is very well quantized and I'm not able to reproduce the quantization myself after "freeing" the model.

(vllm) root@jetson-orin:/tmp# heretic --trust-remote-code true NVIDIA/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
█░█░█▀▀░█▀▄░█▀▀░▀█▀░█░█▀▀  v1.2.0
█▀█░█▀▀░█▀▄░█▀▀░░█░░█░█░░
▀░▀░▀▀▀░▀░▀░▀▀▀░░▀░░▀░▀▀▀  https://github.com/p-e-w/heretic

Detected 1 CUDA device(s) (61.37 GB total VRAM):
* GPU 0: Orin (61.37 GB)

Loading model NVIDIA/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16...
* Trying dtype auto... The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [04:11<00:00, 19.38s/it]
generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 197/197 [00:00<00:00, 500kB/s]
Some parameters are on the meta device because they were offloaded to the cpu.
NemotronH requires an initialized `NemotronHHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.
Ok
* LoRA adapters initialized (targets: out_proj, down_proj, o_proj)
* Transformer model with 52 layers
* Abliterable components:
  * mamba.out_proj: present in 23 layers
  * mlp.down_proj: present in 23 layers
  * attn.o_proj: present in 6 layers

This hybrid model has Mamba/SSM layers. For significantly faster inference, install the optimized kernels:
  pip install causal-conv1d mamba-ssm
  (requires CUDA toolkit ≥ 11.6 — check with nvcc -V; build takes ~10 min)

Resident system RAM: 55.37 GB
Allocated GPU VRAM: 48.34 GB
Reserved GPU VRAM: 49.01 GB

Loading good prompts from mlabonne/harmless_alpaca...
README.md: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 388/388 [00:00<00:00, 959kB/s]
data/train-00000-of-00001.parquet: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 972k/972k [00:01<00:00, 762kB/s]
data/test-00000-of-00001.parquet: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 243k/243k [00:00<00:00, 874kB/s]
Generating train split: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25058/25058 [00:00<00:00, 325729.54 examples/s]
Generating test split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6265/6265 [00:00<00:00, 292685.62 examples/s]
* 400 prompts loaded

Loading bad prompts from mlabonne/harmful_behaviors...
README.md: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 381/381 [00:00<00:00, 905kB/s]
data/train-00000-of-00001.parquet: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15.0k/15.0k [00:00<00:00, 55.2kB/s]
data/test-00000-of-00001.parquet: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.46k/5.46k [00:00<00:00, 19.8kB/s]
Generating train split: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 416/416 [00:00<00:00, 29345.10 examples/s]
Generating test split: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 10006.37 examples/s]
* 400 prompts loaded

Determining optimal batch size...