SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs

👀 TL;DR

SwiReasoning is a training-free method for Pareto-superior reasoning LLMs that dynamically switches between explicit and latent thinking, with a switch count control mechanism to suppress overthinking.

demo.mp4

Comparison of solving the same question with the same reasoning LLM (6s vs. 1min).

🔍 Supported Benchmarks

• Math: GSM8K, MATH500, AIME24, AIME25
• Coding: HumanEval, LeetCode-Contest, MBPP, LiveCodeBench
• General: GPQA Diamond, 2WikiMultihopQA, CommonsenseQA

🔍 Supported Models

• Qwen3, Qwen3.5, and DeepSeek-R1 model families

⚙️ Getting Started

Clone the project

git clone https://github.com/sdc17/SwiReasoning.git
cd SwiReasoning

Environment setup

conda create -n swir python=3.12
conda activate swir
pip install -r requirements.txt
pip install transformers==5.7.0 # Only for Qwen3.5 support

💻 Interactive Chat

python run_chat.py --model_name Qwen/Qwen3-8B --method swir --max_switch_count 2

• Modify --model_name to try different reasoning LLMs.
• Increase --max_switch_count to allow more thinking rounds (default: 2).

Commands:
  exit or q -> [Exit]
  switch <N|none> -> [Set] swir max_switch_count = N (integer >= 1) or None (disabled)
  method <swir|cot|cot_greedy> -> [Set] generation method

• Please check run_chat.sh for more examples.

📈 Evaluation

# Evaluate without switch count control
torchrun --nproc_per_node 1 --nnodes 1 --node_rank 0 --master_port $((RANDOM + 20000)) run.py --model_name Qwen/Qwen3-1.7B \
    --dataset_name gsm8k --batch_size 512 --max_new_tokens 32768 --method swir --alpha 0.6
python merge.py --model_name Qwen/Qwen3-1.7B --dataset_name gsm8k --max_new_tokens 32768 --method swir

# Evaluate with switch count control
torchrun --nproc_per_node 1 --nnodes 1 --node_rank 0 --master_port $((RANDOM + 20000)) run.py --model_name Qwen/Qwen3-8B \
    --dataset_name gsm8k --batch_size 256 --max_new_tokens 32768 --method swir --alpha 0.5 --max_switch_count 2
python merge.py --model_name Qwen/Qwen3-8B --dataset_name gsm8k --max_new_tokens 32768 --method swir

• Increase --nproc_per_node to enable faster evaluation on multiple GPUs.
• Modify --model_name and --dataset_name for evaluation with different models and datasets.
• Please use TOKENIZERS_PARALLELISM=false before torchrun when evaluating on LiveCodeBench.
• Please check run.sh for more examples.

💬 Acknowledgments

We thank the contributors of open-source projects Transformers, Qwen3, and Soft-Thinking.

✨ BibTeX

Please cite if you find our codebase helpful.

@inproceedings{ICLR2026_ddb7bad4,
 author = {Shi, Dachuan and Asi, Abedelkadir and Li, Keying and Yuan, Xiangchi and Pan, Leyan and Lee, Wenke and Xiao, Wen},
 booktitle = {International Conference on Learning Representations},
 editor = {C. Vondrick and B. Hariharan and C. Raffel and L. Pinto and D. Yang and A. Faust},
 pages = {137060--137093},
 title = {SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs},
 url = {https://proceedings.iclr.cc/paper_files/paper/2026/file/ddb7bad46132a323aa9d039000941881-Paper-Conference.pdf},
 volume = {2026},
 year = {2026}
}