INT8 inference issues w8a8

[meven3000]

Hi am am trying to use the Zentorch modules via fp32 with in8 for w8a8, currently i've tried using Quark compile Mistral 7b 0.2 and now also PytorchAO compiled W8a8.

Pytorch W8a8 works as expected without ZenTorch.

Tested from PyTorch ans associated ZenTorch version 2.4 -> 2.8

PyTorch version 2 7 0 (w8a8).txt

basic in8 script.txt

See attached for event log and associated basic testing script.

Noting: assuming again this may be due to enforced AVX512 for int8 (which probably should not be in this case), but certainly no mention of it this time.

[cramasam]

Hi @meven3000 ,

We noticed that in load_quantized_model() (line 83), the INT8 quantized model is being loaded with torch_dtype=torch.float32. Could you please help us with the detail on why the int8 model is loaded as FP32?

Thanks.