Beyond One-Shot Clarification: A Diagnostic Benchmark for Multi-Turn LLM Clarification under Noisy User Feedback

This repository contains the official code and data for the paper "Beyond One-Shot Clarification: A Diagnostic Benchmark for Multi-Turn LLM Clarification under Noisy User Feedback."

🚀 Quick Start

🛠️ Install Requirements

conda create -n ClarifyMT && conda activate ClarifyMT
pip3 install -r requirements.txt

📘 Dataset Generation

export OPENAI_API_KEY=<your_api_key>
export OPENROUTER_API_KEY=<your_api_key>
cd ClarifyMT-Bench
python ./generation/pipeline.py --out_dir <your_path>  --model_name <your_model>

🤖 Model Inference

# local inference
python ./inference/inference_vllm_multiturn.py --model_path <your_local_model_path> --input_file <your_file_path> --output_file <your_file_path>

# API inference 
export OPENROUTER_API_KEY=<your_api_key>
python ./inference/inference_multiturn.py --model_path <your_api_model_path> --input_file <your_file_path> --output_file <your_file_path>

📊 Evaluation

python ./eval/eval.py --input_file <your_file_path> --output_file <your_file_path>

🧠 Run ClarifyAgent

bash ./ClarifyAgent/run_pipeline.sh \
  ./ClarifyMT-Bench/data/data.jsonl \
  ./outputs/clarifyagent \
  <your_local_backbone_path>

ClarifyAgent uses one shared frozen backbone for its five role-specific LLM modules. For every input dialogue turn, it performs the following five forward passes in order:

1. Perceiver extracts task-relevant slots and marks each as filled, unfilled, or conflict.
2. Forecaster predicts the latest user-response condition.
3. Tracker updates the cross-turn slot state using the Perceiver evidence and the previous tracker state.
4. Planner selects clarify or answer.
5. Output generates the user-facing clarification question or final answer.

The pipeline writes the combined result to clarifyagent_results.jsonl and also saves one JSONL file per module for inspection and ablation. The combined output stores the decision in both action and the evaluation-compatible response field, while the user-facing text is stored in final_response. To carry state across rows, give sequential records the same dialogue_id (or conversation_id). A record may instead provide an explicit previous_tracker_state object.

Additional vLLM options can be passed after the backbone path, for example:

bash ./ClarifyAgent/run_pipeline.sh \
  input.jsonl outputs/clarifyagent <your_local_backbone_path> \
  --tensor_parallel_size 2 --max_model_len 8192 --seed 0