GPT-Powered REFramework - Demo###

GPT-Powered REFramework

Introduction

This is a modified version of REFramework that uses OpenAI API to query GPT AI models (GPT-3.5-16k by default) to create, fix and maintain selectors. Please mind, this is an experiment, that shouldn't be implemented on production without a consultation. Selectors are store in applicationModel.json file, so the solution can store and maintain them of the fly. For more details please check the project documentation on GitHub and watch corresponsing video and the demo presentation. https://www.youtube.com/watch?v=m0AbiNFfYG0

Architecture

• Apllication Model.JSON

As I wanted the selectors to be fixed on the fly. I do not want the process to stop just because the selector is wrong and then being fixed by the GPT. I aimed to make the whole process run without stopping. Thus of that, I couldn't use most of what UiPath provides out of the box. The Object Repository is fixed and changing selectors directly in the XAML file would require the solution to be recompiled. Thus I turned into an old-fashioned application model JSON solution. This way I can edit application data on the fly and I can add to JSON additional properties like the name of the UI element that will allow GenAI to recognise the UI element on the webpage based on the plain language description.

Apllication Model.JSON file consist details about the application and list of UI elements described in the plain language. The selector can remain as a placeholder. As I'm suing in the demo full selecotors for each application in the file there is a properly called applicationHeader that determins the highest level selector component. The user needs to set it up for each application manually.

Example of UI element stored in the Application Model.json: "Surname text box": { "uiElementName": "Surname text box", "componentName": "1.1 Insert Ticket", "selector": "" }

The UI element name value needs to be refled in the activity target selector. Example: Type Into - Name text box Target->Selectors -> Glob_JT_ApplicationModel(In_Str_ApplicatinName)("Name text box")("selector").ToString

Please check: 1.1 Insert Ticket.xaml file for more details.

• Process.XAML The next key component of the solution is the changed Process.xmal. As you can see each component is wrapped in the Try-Catch block. When the selector-related exception takes place it triggers FixApplicationModelWithChatGPT workflow that fixes the selectors for the given component. Then the selector generated by the AI is validated and if correct it is saved back to the ApplicationModel.JSON. When all selectors are fixed the process can continue without stopping for maintenance.

• Process configuration The configuration is stored in Config.JSON instead Config Excel file.

Fixing selectors

When the selector-related exception took place then FixApplicationModelWithChatGPT workflow was triggered and fixed all selectors for the given component. It’s being done by sending a query to ChatGPT that consists HTML of the web page, the instruction to GPT so a prompt and the name of the UI element for which the selector needs to be fixed. The template of the prompt you can find in the promptTemplate.txt file in the GPT Templates folder.

As you can see the prompt is quite simple. I'm just asking ChatGPT to generate a UiPath Selecotr for me. I provide some examples and requirements. I'm expecting to receive an output in the JSON format so I can get the selector value from it easily. Then I check if the selector is working and save it back in the applicationModel.json.

But sometimes ChatGPT fails to follow my instructions and the selector is still is invalid. In such a case, the conversation with the AI agent continues by telling the AI that the selector was wrong and asking to generate a new one. The template for that query is in conversationTemplate.txt file. As you can see I'm just giving it some simple guidelines telling AI what could go wrong and how to potentially fix the selector. And usually, it fixes it after the first or second try. For UI elements where I expect some value out, I'm doing additional validation using regex to make sure that I got not only the selector right but the expected value as well.

CI/CD

From th CI/CD perspective the GPT-Powered-REFramework should be run using test or dev enviroment. In theory nothing stops you from running automation and fixing selectors there, but I would advice to follow the same best practice as for solutions maintained by human. So, if there was selector related exception on production the process should be stoped and GPT-Powered-REFramework triggered to run and fix the selectors in ApplicationModel.JSON file. That later can be passed to test and then if everything works correctly, released o production. There is an argument for the Main.xaml file where you can specify if it's a production or non-production run. Please mind that for production run the selector fixing logic will not be triggered.

Other importaint settings

• Config.json

  "GPTMaxTokens" : 500, - Maximum size of the query to the GPT in tokens. Please increase in case dealing with big web pages. "GPTModel" : "gpt-3.5-turbo-16k" - The GPT model to utilize. Dispite demo run using GPT-3.5 Turbo my recommendation is to use GPT4-Turbo, for production ready solutions "GPTRole" : "An expert HTML developer." - Custom instunction for GPT model.

• promptTemplate.txt - The template that stores prompt structure for initial query to GPT model.

• conversationTemplate.txt - The template for guidelines for AI to fix the selector in case it is not correctly generated.

• glob_OpenAI_AIP_Key - Constance that stores Openai API key. https://help.openai.com/en/articles/4936850-where-do-i-find-my-api-key https://openai.com/pricing

If you want to run the demos

• MAIN Glob_JT_ApplicationModel - The global variable that stores the application model. Glob_Dic_Config - Global variable that stores configuration. in_Bool_UseConfigJSON - A flag controlling from where the configuration should be obtained. By default "true", so the configuration will be obtained from Config.json. If false configuration will be obtained from Config.xlsx file. Config.xlsx file is not setup to work with the demo. in_Bool_ProductionRun - if True, selector fixing logic will not be triggered.

• Queue items for the process you can find in the /Data catalogue. You can import them directly to your Orchestrator.

• MAIN.XAML/PROCESS.XAML

bool_saveSelecotrs - FOR A PURPOSE OF THE DEMO. SELECTORS ARE NOT SAVED IN THE APPLICATION JSON FILE. REMEMBER TO CHANGE THE bool_saveSelectors FLAG TO 'TRUE' IF YOU WANT TO USE IT BEYOND THE DEMO. I recommend to bool_saveSelecotrs as false if you just want to run selectors generation demos.