Query about Error Information and stack traces

[ramsey-coding]

I am unsure about what you are referring to with Error Information and stack traces in this paper.

Error Information: Are you referring to errors in the code generated by the Large Language Model (LLM) that result in syntax errors when you try to compile or execute the code?

Stack Trace: How are stack traces obtained for this project? Do you mean that the code generated by the LLM is causing exceptions, and you are capturing the stack traces when these exceptions are thrown during execution?

Could you please provide more details or clarify these points?

[ramsey-coding]

Also how do you collect runtime value other than instrumentation?

[norhh]

https://github.com/PyRepair/maniple/blob/master/data/BGP32/black/10/110111101/prompt.md
You can look at an example prompt here. The error information is the error obtained when executing the test. It consists of the error message which is the AssertionError, and the stacktrace pointing to the location of the error.

[norhh]

The runtime information can be collected by instrumenting the code which stores the values of variables at the start and end of the function and prints it into a file.

[ramsey-coding]

how do you do code instrumentation? Is not it costly? Do you modify the code?

[norhh]

It is cheap for python. It might be a bit expensive for languages like C++ (linking might take some time), but it will be a one time execution.
Maniple performs instrumentation by modifying AST.

[ramsey-coding]

Do you share instrumentation code in the artefact?

[norhh]

It is in this repository: https://github.com/PyRepair/pyr_benchmark_wrangling