Missing documentation - heretic hardware requirements

[dagbdagb]

Hi.

2 things I struggle to figure out on my own:

gpt-oss-120b and its smaller sibling come with weights in mxfp4 format.
Can I use Ampere hardware (rtx 3090) to decensor gpt-oss-120b? Cuda arch 86, so no native support for mxfp4. Does that matter?

Also, what are the hard memory requirements for successfully decensoring gpt-oss-120b? Is there a trivial way to calculate that? I see #83 , but still fail to grasp if less memory just means the process takes longer, or if it will fail.

[accemlcc]

hey

Hardware Requirements for GPT-OSS-120B (MXFP4)

Short answer: No, RTX 3090 won't work. You need Hopper GPUs (H100) or newer.

Why RTX 3090 doesn't work:

• MXFP4 requires Compute Capability ≥ 9.0 for native support
• RTX 3090 has CC 8.6 (Ampere)
• While transformers CAN dequantize to bf16, that defeats the purpose and requires massive VRAM

---

Anyways, it's already done and looks good:

https://huggingface.co/kldzj/gpt-oss-120b-heretic-v2
https://huggingface.co/bartowski/kldzj_gpt-oss-120b-heretic-v2-GGUF

[dagbdagb]

hey

Hardware Requirements for GPT-OSS-120B (MXFP4)

Short answer: No, RTX 3090 won't work. You need Hopper GPUs (H100) or newer.

Why RTX 3090 doesn't work:

• MXFP4 requires Compute Capability ≥ 9.0 for native support
• RTX 3090 has CC 8.6 (Ampere)
• While transformers CAN dequantize to bf16, that defeats the purpose and requires massive VRAM

Anyways, it's already done and looks good:

https://huggingface.co/kldzj/gpt-oss-120b-heretic-v2 https://huggingface.co/bartowski/kldzj_gpt-oss-120b-heretic-v2-GGUF

The timestamp of those files suggest that they are missing a few fixes in heretic 1.1? Happy to be set straight if this is incorrect.

[accemlcc]

The timestamp of those files suggest that they are missing a few fixes in heretic 1.1? Happy to be set straight if this is incorrect.

Oops... you're right, they were made with the old version of Heretic. I was misled by V2, mea culpa.

[accemlcc]

If you still want to try it yourself: you would need at least two H100s, PyTorch 2.8, the latest Triton version and my #60, which also circumvents the 3D tensor problem.

[dagbdagb]

Might have to look into how to rent hardware and stuff. Unlikely to happen while the calendar still says 2025.

[dagbdagb]

@accemlcc
At https://docs.unsloth.ai/models/gpt-oss-how-to-run-and-fine-tune , I find this little nugget:

"MXFP4 is not actually supported on Ampere and older GPUs, so Triton provides tl.dot_scaled for MXFP4 matrix multiplication. It upcasts the matrices to BF16 internaly on the fly."

Is this something which can be employed by heretic?

[accemlcc]

@accemlcc At https://docs.unsloth.ai/models/gpt-oss-how-to-run-and-fine-tune , I find this little nugget:

"MXFP4 is not actually supported on Ampere and older GPUs, so Triton provides tl.dot_scaled for MXFP4 matrix multiplication. It upcasts the matrices to BF16 internaly on the fly."

Is this something which can be employed by heretic?

For clarification: Tools like LM Studio use GGUF, which is highly optimized for split CPU/GPU inference.

Heretic uses native PyTorch/Transformers.

More importantly: Heretic doesn't just generate text; it extracts hidden states (residuals) from the model to calculate vectors. This requires significantly more VRAM bandwidth and capacity than simple chat inference.

So while you might get 5 t/s chatting in GGUF, running an optimization loop in PyTorch with residual extraction on a single 3090 will likely result in OOMs or extreme slowness due to PCIe bottlenecks.

[dagbdagb]

@accemlcc At https://docs.unsloth.ai/models/gpt-oss-how-to-run-and-fine-tune , I find this little nugget:
"MXFP4 is not actually supported on Ampere and older GPUs, so Triton provides tl.dot_scaled for MXFP4 matrix multiplication. It upcasts the matrices to BF16 internaly on the fly."
Is this something which can be employed by heretic?

For clarification: Tools like LM Studio use GGUF, which is highly optimized for split CPU/GPU inference.

Heretic uses native PyTorch/Transformers.

More importantly: Heretic doesn't just generate text; it extracts hidden states (residuals) from the model to calculate vectors. This requires significantly more VRAM bandwidth and capacity than simple chat inference.

So while you might get 5 t/s chatting in GGUF, running an optimization loop in PyTorch with residual extraction on a single 3090 will likely result in OOMs or extreme slowness due to PCIe bottlenecks.

Triton permits extending pytorch's 'native' features. And, in the case of unsloth, casting between MXFP4 and FP16 (and FP32) as required by the hardware at hand.

I do have multiple 3090s, but all connected either via TB3 or PCIe 4x. If system/GPU bandwidth is important for the performance of heretic, I accept it will be slow. Unlike for inferencing, where my setup regularly provides 110-120 t/s with an empty context. (gpt-oss-120b-UD-Q8_K_XL, all layers on GPU, potato PC)

So, is system/GPU bandwidth important for heretic? Moreso than compute performance or GPU memory bandwidth? Or would up/down-casting be a/the killer for performance?

What is the expected runtime for heretic with gpt-oss-120b with a single h200? Or, if 2 are required due to VRAM requirements, with two? Does performance scale somewhat linearly?

Despite being technically competent in many subjects, I find it hard to achieve anything but very superficial knowledge about LLMs/pytorch/transformer/triton. Too much stuff already out there, and the fire-hose runs at full blast with new stuff 24x7. I very much appreciate the patience exhibited explaining this.

@p-e-w
I suggest a little blurb is added to the default README about hardware requirements/suggestions and a guesstimate for approximate runtime.

[accemlcc]

@dagbdagb

Okay, that context helps.

Regarding the Unsloth/Triton integration: That is a significant architectural decision at @p-e-w 's discretion. I will not be implementing it.

Regarding the H200: A full run should take less than 1 hour.

[p-e-w]

Heretic uses Transformers to perform inference, so Heretic's hardware requirements are exactly the same as those for running the model with Transformers. I have no intention to add more details to the README regarding that, as that would replicate information that is already available all over the web and can be highly specific to individual setups.

What is the expected runtime for heretic with gpt-oss-120b with a single h200?

That depends on the configuration (prompt datasets, n_trials, max_response_length etc.). Heretic estimates remaining runtime during optimization, so you can just try it and it will tell you.

[dagbdagb]

Heretic uses Transformers to perform inference, so Heretic's hardware requirements are exactly the same as those for running the model with Transformers. I have no intention to add more details to the README regarding that, as that would replicate information that is already available all over the web and can be highly specific to individual setups.

What is the expected runtime for heretic with gpt-oss-120b with a single h200?

That depends on the configuration (prompt datasets, n_trials, max_response_length etc.). Heretic estimates remaining runtime during optimization, so you can just try it and it will tell you.

Thank you. And thank you for heretic.

[dagbdagb]

Happy to report that heretic 1.1 did the job on gpt-oss-20b with 4x RTX3090, despite the lack of hw-support for mxfp4.

Elapsed time: 2h 6m
Refusals: 24/100, KL divergence: 0.1771

Had to set export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

Edit:
The saved safetensors weigh in at 39.0GB total (original: 11.9GB). I assume pytorch has converted the original mxfp4 to f16 on the fly, or something along that line.

[p-e-w]

@dagbdagb

Had to set export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

This is already set on startup:

heretic/src/heretic/main.py

Lines 122 to 127 in 02a5237

	# Enable expandable segments to reduce memory fragmentation on multi-GPU setups.
	if (
	"PYTORCH_ALLOC_CONF" not in os.environ
	and "PYTORCH_CUDA_ALLOC_CONF" not in os.environ
	):
	os.environ["PYTORCH_ALLOC_CONF"] = "expandable_segments:True"

Are you sure that was necessary to set again manually? Because that would be a bug.

[dagbdagb]

I am quite sure that was the only change I did between a failed run and a successful run, yes. Should your and be an or?