Request for Reproduction Instructions for NAACL 2025 Paper Experiments

[zhiyuan-ning]

Hi authors,

Thank you for releasing the code for your NAACL 2025 paper, it's an exciting and timely contribution!

After reading through the README.md and exploring the repository, I’m having trouble understanding how to reproduce the experiments reported in the paper. In particular, it would be really helpful to have:

1. Step-by-step instructions or commands for reproducing the main results/figures/tables from the paper.
2. Expected input/output formats for main.py — what arguments (if any) does it take, and what should we expect as output?
3. An example run or configuration file, if applicable, to help illustrate the full workflow.
4. Clarification on Docker usage: is Docker required for reproducing the paper results, or is it just an optional setup utility?
5. Details on external dependencies, such as datasets, model APIs (e.g., OpenAI, Claude, etc.), or keys that need to be set up beforehand.

If you could provide a minimal example (e.g., one command and its expected output), that would go a long way in helping the community reproduce and build upon your work.

Thanks again for sharing your code — looking forward to your guidance!

Best regards,
Zhiyuan Ning

[RunhuiWang]

Hi Zhiyuan,

Thanks for your interests in our work and constructive suggestions! We will provide more details and update the README soon.

As for the docker usage, it is recommended because:

1. it makes sure that each run is independent and does not interfere with other experiments
2. it reduces the risks of running critical bash commands that might affect the whole system, especially when the server is shared by multiple users.

Best,
Runhui

[zhiyuan-ning]

Thanks again, Runhui — really appreciate your response and the clarification about Docker.

As a quick follow-up to my earlier question: while I understand that a full README update is in progress, I’d like to ask a bit more specifically about how to reproduce the experimental results reported in your NAACL 2025 paper.

More concretely:

• Could you share an example command or workflow that reproduces one of the paper’s reported results (e.g., from a specific table or figure)?
• How were the evaluation metrics computed — are there scripts or logs available for that?
• Would it be possible to release the complete agent execution traces from your experiments — including action sequences, intermediate observations, model responses, and final outputs?

Having access to the full trajectories would be extremely valuable for the research community. It would allow others to:

• analyze agent behavior and reasoning strategies in depth,
• better understand failure cases and design limitations,
• and build future methods grounded in reproducible and interpretable evidence.

Since CSR-Bench is designed as a benchmark for LLM agents, comprehensive transparency like this would greatly enhance its long-term value and impact.

Thanks again for your work — looking forward to the updated release, and happy to provide feedback!

Best,
Zhiyuan

[sreekarsashaankadibhatla-snow]

I can't see any docker_setup.sh.
I'm trying to use this to benchmark the latest models. What would be your suggestion? @RunhuiWang . It would help me to proceed with benchmarking my usecase. of open source models vs latest models like gpt-5, claude-4.5 too