[Bug]: eval bug with tied lm-head and act_bits<=8

[wenhuach21]

Problem Description

~

Reproduction Steps

/models/Qwen3-0.6B
--format
"fake"
--bits
2
--group_size
16
--nsamples
1
--tasks
piqa
--iters
0
--disable_opt_rtn
--device_map
"auto"
--act_bits
8

Environment Information

No response

Error Logs

Additional Context

Running loglikelihood requests: 0%| | 0/3676 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/wenhuach/auto-round/auto_round/main.py", line 799, in
run()
~~~^^
File "/home/wenhuach/auto-round/auto_round/main.py", line 783, in run
start()
~~~~~^^
File "/home/wenhuach/auto-round/auto_round/main.py", line 516, in start
tune(args)
~~~~^^^^^^
File "/home/wenhuach/auto-round/auto_round/main.py", line 729, in tune
run_model_evaluation(model, tokenizer, autoround, folders, formats, device_str, args)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wenhuach/auto-round/auto_round/eval/evaluation.py", line 433, in run_model_evaluation
evaluate_with_model_instance(model, tokenizer, device_str, args)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wenhuach/auto-round/auto_round/eval/evaluation.py", line 277, in evaluate_with_model_instance
res = simple_evaluate_user_model(
model,
...<6 lines>...
add_bos_token=args.add_bos_token,
)
File "/home/wenhuach/auto-round/auto_round/eval/evaluation.py", line 60, in simple_evaluate_user_model
return lm_eval.simple_evaluate(
~~~~~~~~~~~~~~~~~~~~~~~^
model=hflm, model_args=None, batch_size=batch_size, max_batch_size=max_batch_size, limit=limit, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/lm_eval/utils.py", line 458, in _wrapper
return fn(*args, **kwargs)
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/lm_eval/evaluator.py", line 357, in simple_evaluate
results = evaluate(
lm=lm,
...<12 lines>...
confirm_run_unsafe_code=confirm_run_unsafe_code,
)
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/lm_eval/utils.py", line 458, in _wrapper
return fn(*args, **kwargs)
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/lm_eval/evaluator.py", line 585, in evaluate
resps = getattr(lm, reqtype)(cloned_reqs)
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/lm_eval/api/model.py", line 453, in loglikelihood
return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/lm_eval/models/huggingface.py", line 1296, in _loglikelihood_tokens
self._model_call(batched_inps, **call_kwargs),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/lm_eval/models/huggingface.py", line 965, in _model_call
return self.model(inps).logits
~~~~~~~~~~^^^^^^
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/transformers/utils/generic.py", line 918, in wrapper
output = func(self, *args, **kwargs)
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 494, in forward
logits = self.lm_head(hidden_states[:, slice_indices, :])
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/wenhuach/miniforge3/envs/autoround/lib/python3.13/site-packages/torch/nn/modules/linear.py", line 134, in forward
return F.linear(input, self.weight, self.bias)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Inference tensors do not track version counter.

[lvliang-intel]

@wenhuach21,
I can't reproduce your issue using latest main branch code.

CUDA_VISIBLE_DEVICES=7 auto_round Qwen/Qwen3-0.6B --format fake --bits 2 --group_size 16 --nsamples 1 --tasks piqa --iters 0 --disable_opt_rtn --device_map auto --act_bits 8
2026-02-10 15:34:11 INFO main.py L574: start to quantize Qwen/Qwen3-0.6B
Loading weights: 100%|██████| 311/311 [00:00<00:00, 988.59it/s, Materializing param=model.norm.weight]
The tied weights mapping and config for this model specifies to tie model.embed_tokens.weight to lm_head.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with tie_word_embeddings=False to silence this warning
2026-02-10 15:34:44 INFO base.py L709: Activation adopting weight data_type: int
2026-02-10 15:34:45 WARNING base.py L3270: amp is set to FALSE as the current cpu device does not support the 'bf16' data type.
2026-02-10 15:34:45 INFO base.py L486: using torch.float32 for quantization tuning
2026-02-10 15:34:45 WARNING base.py L836: activation quantization is an experimental feature with limited support and a complex API. And please save the quantized model to fake format as real deployment is not supported currently
2026-02-10 15:34:45 WARNING base.py L895: for bits <= 2, it is recommended to enable auto-round-best and turn on --enable_alg_ext
2026-02-10 15:34:45 WARNING base.py L1639: low_cpu_mem_usage is only supported when immediate_packing is True. Setting low_cpu_mem_usage to False.
Quantizing model.layers.0.mlp.down_proj: 3%|▋ | 6/196 [00:00<00:00, 234.98it/s]2026-02-10 15:34:45 INFO device.py L1468: 'peak_ram': 3.73GB
Quantizing model.layers.1.mlp.down_proj: 7%|█▌ | 13/196 [00:00<00:03, 52.51it/s]2026-02-10 15:34:45 INFO device.py L1468: 'peak_ram': 3.73GB
Quantizing model.layers.2.mlp.down_proj: 10%|██▎ | 20/196 [00:00<00:03, 48.52it/s]2026-02-10 15:34:45 INFO device.py L1468: 'peak_ram': 3.74GB
Quantizing model.layers.3.mlp.down_proj: 14%|███▏ | 27/196 [00:00<00:03, 47.09it/s]2026-02-10 15:34:46 INFO device.py L1468: 'peak_ram': 3.74GB
Quantizing model.layers.4.mlp.down_proj: 17%|███▉ | 34/196 [00:00<00:03, 46.44it/s]2026-02-10 15:34:46 INFO device.py L1468: 'peak_ram': 3.74GB
Quantizing model.layers.5.mlp.down_proj: 21%|████▊ | 41/196 [00:00<00:03, 46.09it/s]2026-02-10 15:34:46 INFO device.py L1468: 'peak_ram': 3.74GB
Quantizing model.layers.6.mlp.down_proj: 24%|█████▋ | 48/196 [00:00<00:03, 45.56it/s]2026-02-10 15:34:46 INFO device.py L1468: 'peak_ram': 3.74GB
Quantizing model.layers.7.mlp.down_proj: 28%|██████▍ | 55/196 [00:01<00:03, 45.32it/s]2026-02-10 15:34:46 INFO device.py L1468: 'peak_ram': 3.75GB
Quantizing model.layers.8.mlp.down_proj: 32%|███████▎ | 62/196 [00:01<00:02, 45.82it/s]2026-02-10 15:34:46 INFO device.py L1468: 'peak_ram': 3.75GB
Quantizing model.layers.9.mlp.down_proj: 35%|████████ | 69/196 [00:01<00:02, 46.20it/s]2026-02-10 15:34:46 INFO device.py L1468: 'peak_ram': 3.75GB
Quantizing model.layers.10.mlp.down_proj: 39%|████████▌ | 76/196 [00:01<00:02, 46.42it/s]2026-02-10 15:34:47 INFO device.py L1468: 'peak_ram': 3.75GB
Quantizing model.layers.11.mlp.down_proj: 42%|█████████▎ | 83/196 [00:01<00:02, 46.60it/s]2026-02-10 15:34:47 INFO device.py L1468: 'peak_ram': 3.76GB
Quantizing model.layers.12.mlp.down_proj: 46%|██████████ | 90/196 [00:01<00:02, 46.73it/s]2026-02-10 15:34:47 INFO device.py L1468: 'peak_ram': 3.76GB
Quantizing model.layers.13.mlp.down_proj: 49%|██████████▉ | 97/196 [00:01<00:02, 46.67it/s]2026-02-10 15:34:47 INFO device.py L1468: 'peak_ram': 3.76GB
Quantizing model.layers.14.mlp.down_proj: 53%|███████████▏ | 104/196 [00:02<00:01, 46.46it/s]2026-02-10 15:34:47 INFO device.py L1468: 'peak_ram': 3.76GB
Quantizing model.layers.15.mlp.down_proj: 57%|███████████▉ | 111/196 [00:02<00:01, 46.36it/s]2026-02-10 15:34:47 INFO device.py L1468: 'peak_ram': 3.76GB
Quantizing model.layers.16.mlp.down_proj: 60%|████████████▋ | 118/196 [00:02<00:01, 46.44it/s]2026-02-10 15:34:48 INFO device.py L1468: 'peak_ram': 3.77GB
Quantizing model.layers.17.mlp.down_proj: 64%|█████████████▍ | 125/196 [00:02<00:01, 44.86it/s]2026-02-10 15:34:48 INFO device.py L1468: 'peak_ram': 3.77GB
Quantizing model.layers.18.mlp.down_proj: 67%|██████████████▏ | 132/196 [00:02<00:01, 45.38it/s]2026-02-10 15:34:48 INFO device.py L1468: 'peak_ram': 3.77GB
Quantizing model.layers.19.mlp.down_proj: 71%|██████████████▉ | 139/196 [00:02<00:01, 45.76it/s]2026-02-10 15:34:48 INFO device.py L1468: 'peak_ram': 3.77GB
Quantizing model.layers.20.mlp.down_proj: 74%|███████████████▋ | 146/196 [00:03<00:01, 45.95it/s]2026-02-10 15:34:48 INFO device.py L1468: 'peak_ram': 3.77GB
Quantizing model.layers.21.mlp.down_proj: 78%|████████████████▍ | 153/196 [00:03<00:00, 46.00it/s]2026-02-10 15:34:48 INFO device.py L1468: 'peak_ram': 3.78GB
Quantizing model.layers.22.mlp.down_proj: 82%|█████████████████▏ | 160/196 [00:03<00:00, 46.08it/s]2026-02-10 15:34:48 INFO device.py L1468: 'peak_ram': 3.78GB
Quantizing model.layers.23.mlp.down_proj: 85%|█████████████████▉ | 167/196 [00:03<00:00, 46.18it/s]2026-02-10 15:34:49 INFO device.py L1468: 'peak_ram': 3.78GB
Quantizing model.layers.24.mlp.down_proj: 89%|██████████████████▋ | 174/196 [00:03<00:00, 46.33it/s]2026-02-10 15:34:49 INFO device.py L1468: 'peak_ram': 3.78GB
Quantizing model.layers.25.mlp.down_proj: 92%|███████████████████▍ | 181/196 [00:03<00:00, 46.44it/s]2026-02-10 15:34:49 INFO device.py L1468: 'peak_ram': 3.79GB
Quantizing model.layers.26.mlp.down_proj: 96%|████████████████████▏| 188/196 [00:03<00:00, 46.39it/s]2026-02-10 15:34:49 INFO device.py L1468: 'peak_ram': 3.79GB
Quantizing model.layers.27.mlp.down_proj: 99%|████████████████████▉| 195/196 [00:04<00:00, 46.45it/s]2026-02-10 15:34:49 INFO device.py L1468: 'peak_ram': 3.79GB
Quantizing model.layers.27.mlp.down_proj: 100%|█████████████████████| 196/196 [00:04<00:00, 45.45it/s]
2026-02-10 15:34:49 WARNING base.py L3204: Support for exporting activation quantization is limited. Please ensure that your configuration is supported.
Writing model shards: 100%|█████████████████████████████████████████████| 1/1 [00:03<00:00, 3.51s/it]
2026-02-10 15:34:53 WARNING formats.py L325: Skipping source model Python file copy due to error: cannot access local variable 'cache_dir' where it is not associated with a value
2026-02-10 15:34:53 INFO device.py L1468: 'peak_ram': 3.79GB
2026-02-10 15:34:53 INFO evaluation.py L411: Using lm-eval version 0.4.11.dev0
2026-02-10 15:34:53 WARNING evaluation.py L273: This API does not support auto currently, reset eval_bs to 16
pretrained model kwarg is not of type str. Many other model arguments may be ignored. Please do not launch via accelerate or use parallelize=True if passing an existing model this way.
Passed an already-initialized model through pretrained, assuming single-process call to evaluate() or custom distributed integration
piqa_train.parquet: 100%|████████████████████████████████████████| 2.64M/2.64M [00:01<00:00, 1.58MB/s]
piqa_validation.parquet: 100%|█████████████████████████████████████| 300k/300k [00:00<00:00, 7.48MB/s]
piqa_test.parquet: 100%|███████████████████████████████████████████| 496k/496k [00:00<00:00, 17.0MB/s]
Generating train split: 16113 examples [00:00, 1590745.45 examples/s]
Generating validation split: 1838 examples [00:00, 754465.72 examples/s]
Generating test split: 3084 examples [00:00, 1123532.84 examples/s]
100%|███████████████████████████████████████████████████████████| 1838/1838 [00:01<00:00, 1095.85it/s]
Running loglikelihood requests: 100%|█████████████████████████████| 3676/3676 [02:52<00:00, 21.36it/s]

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
piqa	1	none	0	acc	↑	0.5517	±	0.0116
		none	0	acc_norm	↑	0.5397	±	0.0116

evaluation running time=206s

[lvliang-intel]

@wenhuach21,
Is this issue still reproducible with the latest code? If not, we will close this issue.