Detailed evaluation setup for VerilogEval

[maekawatoshiki]

Hi.

From your preprint, Nexus achieved 100% accuracy on non-self-verifying VerilogEval V1 Human (≒ VerilogEval V2).
I'd like to reproduce the result, but the only script I found is eda.py, which I believe was not used for the evaluation described in the paper.

Could you share the full evaluation setup for VerilogEval?

[humza-sami]

Hi @maekawatoshiki,

Yes, eda.py is an example script and was not used for the specific VerilogEval evaluation reported in the paper. We have a separate repository that contains the full evaluation and benchmarking workflow. While this repository hasn't been released yet, we are planning to make it publicly available soon.
In the meantime, if you're looking to reproduce the results or need assistance setting up the evaluation, I’d be happy to help. Feel free to reach out to me directly at humza.sami@primis.ai, and we can coordinate the setup together.

[maekawatoshiki]

Thank you for your quick response. I've sent you an email; let's continue the conversation there.

[a265589]

Hi @humza-sami ,
I also want to reproduce the results of verilogEval Human. I've sent you an email. I'm not sure if you see the mail.

[1347377011]

Hi @humza-sami
I also sent you an email about the reproduction of verilogeval, but the email was rejected. Could you provide another email address? Or could you contact me? (My email address is leslie93726@gmail.com)
Thank you very much!