Replicate the evaluations from the paper.

[fmantianxing]

Hello，

1.In the paper，DeepSeek-R1's call Accuracy

2.But, i got 37.9，when i run the code: python3 EVAL/eval_T/0_call_acc.py --source LLM_generated/Bench_T_general_purpose/output_DeepSeek-R1.jsonl --target output/deepseek --GPUs 0,1,2,3

Does this replication process look correct to you?
Please reply to me

Thinks

[Schopenhauer-loves-Hegel]

same problem, my command is:

python EVAL/eval_T/0_call_acc.py --source LLM_generated/Bench_T_general_purpose/output_DeepSeek-R1.jsonl --target results/batch_test --GPUs 0

got this result:

Correct execution rate: 38.55%
['sigmoid_conv2d.py', 'sigmoid_argmax.py', 'svd.py', 'dropout_relu_batch_norm_conv2d.py', 'add.py', 'relu.py', 'fused_embedding_add_tanh.py', 'fused_mv_sigmoid_sub.py', 'fused_cosine_embedding_loss_with_normalization.py', 'fused_transformer_block.py', 'log1p.py', 'sigmoid_batch_norm.py', 'symmetric_matrix_vector_norm.py', 'fused_svd_reconstruct.py', 'selu.py', 'scaled_add_norm.py', 'leaky_relu_conv2d.py', 'cos_avg_pool1d.py', 'sum_std.py', 'mul_relu.py', 'gelu_conv2d.py', 'fused_fractional_max_pool2d_with_relu.py', 'scaled_add_dot.py', 'relu_batch_norm_conv2d.py', 'fused_cholesky_solve.py', 'fused_gather_masked_fill.py', 'addmm.py', 'exp.py', 'matrix_multiply_symmetric.py', 'fused_hardshrink_dropout.py', 'tensordot_rsqrt.py', 'softmax_log.py', 'dropout_sigmoid_linear.py', 'bitwise_and.py', 'gelu_std.py', 'digamma.py', 'softmax_mul.py', 'lu.py', 'pseudoinverse_svd.py', 'exp_mean.py', 'low_rank_svd_approximation.py', 'symmetric_mm_and_abs_sum.py', 'logspace.py', 'solve_and_add_scaled_vector.py', 'pixel_shuffle_conv2d.py', 'relu_max_pool2d_conv2d.py', 'erf.py', 'det.py', 'cholesky_solve.py', 'abs.py', 'leaky_relu.py', 'invert_matrix_lu.py', 'std.py', 'tril_mm_and_scale.py', 'airy_ai.py', 'elu_linear.py', 'Adam.py', 'broadcast_tensors.py', 'relu_conv2d.py', 'log.py', 'conv2d_add.py', 'ones_like.py', 'autocast.py', 'fftn.py']
Above is call test for output_DeepSeek-R1