Fix code generation so that it no longer errors when generating on top of previously generated code

[jasonmreding]

Description

This PR updates code generation to avoid errors after making changes to a proto file and regenerating over previously generated code.

• Fixes Not able to regenerate client code after changing .proto file #494
• Fixes Server fails to rebuild if adding new rpc to .proto file #480
• Fixes gPRC Template Generation Utility fails #365
• Fixes Regenerating .proto results in error (since v.1.0.0.6) #346
• Fixes Regenerating helloworld.proto doesn't generate client VI for new RPC #92

Implementation / Design

The main issue was that when generating oneof converters from a template, we previously attempted to save over the previously generated VI on disk using the newly scripted template VI. This would generate an error because the previously generated VI would get loaded into memory when the previously generated library was loaded into memory, and LV doesn't let you save a VI using the same path as another VI loaded into memory. I tried various save as, rename, or save to temp file and then copy over file on disk approaches. However, for reasons I cannot entirely explain, they all either still suffered from the same save error or resulted in broken generated code since the generated VIs no longer though they belonged to the library. Ultimately, I settled on an approach where I just replace the contents of the existing VI with the contents from the template instead. The changes for this are included as part of the first commit.

There were also some lingering issues created from VI reference leaks keeping VIs loaded into memory longer than intended. This was also leading to name collisions during save in some instances. These were identified using Desktop Execution Trace Toolkit, and the subsequent changes are part of the second commit.

Testing

I manually tested this using the route_guide.proto file from the examples and was able to successfully regenerate both the client and server after making changes to the proto file to add a new RPC. I also tested helloworld.proto following the steps in the getting started guide. While testing, I still observed the following problems:

• If the server code contains user code which implements an RPC using one of the generated typedefs on the diagram, the final generated result might not be runnable as is since the typedefs are rebuilt from scratch as part of the generation, and LV doesn't always preserve existing cluster item mappings as the typedef is updated multiple times through the scripting process. This seemed to mostly be a problem with nested typedefs. If you manually configure the typedef instance to not Auto-Update from Type Def., then the problem goes away. In general, I think we should encourage service authors to implement all of the server logic in a second library. I believe this would avoid this issue as well.
• Removing RPCs from the proto file and regenerating the server code doesn't seem to actually remove the events from the server. It's questionable if we want to automatically remove event cases that might contain user code from the event structure. However, updating the event list and the RPC event structures spawned from Start Async.vi seems like it would make sense.
• Regeneration of proto files containing oneof fields still doesn't work. The scripting code for oneof fields always expects to find VI names provided by the template before they are renamed based on the field names from the proto file.

Since these are existing issues and a number of users seem to be hindered by lack of any code generation on top of existing code, I would like to move forward and address the other issues at a later date.

[Madd3DScientist]

I pulled down the code, but could not make the VIP packages. The build.py script uses distutils, which was deprecated in Python 3.10 and removed in Python 3.12. I currently have 3.13. Migration advice for the two specific modules used is to re-implement the functionality yourself. I can take a look at that, if you would like.

[jasonmreding]

I pulled down the code, but could not make the VIP packages. The build.py script uses distutils, which was deprecated in Python 3.10 and removed in Python 3.12. I currently have 3.13. Migration advice for the two specific modules used is to re-implement the functionality yourself. I can take a look at that, if you would like.

You are welcome to take a look at it if you like. In the meantime, you can also test the generator changes by running it from source rather than installing it through a VI package. For this to work, you can simply copy the existing dlls installed at vi.lib\gRPC\gRPC Server and Client Template [2]\gRPC Scripting Tools\Proto Parser API\Libraries to the same relative location under the root directory for the repo (labview source\Client Server Support New\gRPC Scripting Tools\Proto Parser API\Libraries). Just remember to clean them up when you are done so you don't accidentally submit them to your branch later.

[Madd3DScientist]

I replaced the distutils with their shutil equivalents and the python script worked, although I did need to specify the LabVIEW port, as the script cratered on a binary byte in my LV ini file. I tested in LV2025Pro. The generation of a complex project with an added rpc command appeared to work. I got a new frame in the event structure and the existing code was still there. I also fully tested a smaller project and everything worked.

The typedef issue plagues just about all large LabVIEW projects. A bottom-up recursive force compile of all controls and VIs in the project will often fix the issue. In rarer occasions, you need to track down the offending VI and manually replace the bad control. Most experienced LV large application developers have a VI to do this.

If you would like me to submit a pull request with the script changes, let me know.

[jasonmreding]

If you would like me to submit a pull request with the script changes, let me know.

I assume you are talking about the python script where you replaced distutils with shutil. Yes you are welcome to submit a PR with updates to the script. I don't think it is on anyone else's radar at the moment.

[yash-ni]

The changes look good to me. I still suspect that we are leaking a reference somewhere, which is why we are unable to replace that particular VI. In any case, this should unblock the protofile regeneration issue.

[jasonmreding]

The changes look good to me.

If you are good with the changes and don't have any other comments, can you approve the PR and I'll submit.

I still suspect that we are leaking a reference somewhere, which is why we are unable to replace that particular VI. In any case, this should unblock the protofile regeneration issue.

I think it has more to do with the fact that once the library itself is loaded, it seems to immediately load some VIs in the library and keep them loaded. It's also possible that some of the leaks I cleaned up after the fact would have allowed the original code to work as expected. Regardless, there are still a number of issues with the code generation. and this is just a band-aid to get past the most serious issues for the majority of people. If you use oneof fields in your API, regeneration will still fail for other reasons, and we also do a poor job of cleaning up/deleting code if you were to rename or delete things from the proto file. To fix these issues, I think the next pass of the generator should always generate code from scratch in a temp library and then copy files to the desired project after the fact. That should simplify the code generation, provide better consistency of results, and more cleanly handle deletes and renames.