Automation

src/prerna/project/api/IProject.java

@@ -60,7 +60,7 @@ public interface IProject extends IEngine, IMCP {
 	String NOTEBOOK_FOLDER = ".notebooks";
 
 	enum PROJECT_TYPE {
-		BLOCKS, CODE, WORKSPACE, SKILL, INSIGHTS, NOTEBOOK,
+		BLOCKS, CODE, WORKSPACE, SKILL, INSIGHTS, NOTEBOOK, AUTOMATION,
 	};
 
 	/**

src/prerna/reactor/automation/AGENTS.md

@@ -0,0 +1,140 @@
+# Automation Engine — Agent Guide
+
+Executes sequential node pipelines against SEMOSS engines. Users build a pipeline in the form editor (FE), save it as `automation.json`, then trigger it manually. Each run is tracked in the DB and the FE polls for progress.
+
+## How it works
+
+```
+FE calls runPixelAsync("TriggerAutomation(...)")
+  → Monolith spawns a virtual thread, returns jobId immediately
+  → FE polls GetActiveAutomationRun every 500ms (up to 10×) to get runId
+
+TriggerAutomationReactor (virtual thread, synchronous)
+  → reads automation.json (nodes in order)
+  → claims single-run slot (AUTOMATION_ACTIVE_RUN) → runId visible to FE
+  → inserts AUTOMATION_RUNS + AUTOMATION_NODE_OUTPUTS rows
+  → calls AutomationRunEngine.run() synchronously
+  → returns completed run result to jobId slot when done
+
+AutomationRunEngine (same virtual thread)
+  → iterates nodes in saved order
+  → dispatches each node to its IAutomationNodeExecutor
+  → writes node output + status to AUTOMATION_NODE_OUTPUTS after each node
+  → FE polls GetAutomationRun every 3s while runId is known
+```
+
+## Reactors
+
+| Reactor | Pixel | What it does |
+| --- | --- | --- |
+| `TriggerAutomationReactor` | `TriggerAutomation(project=["id"])` | Starts a run synchronously; returns completed run result |
+| `GetActiveAutomationRunReactor` | `GetActiveAutomationRun(project=["id"])` | Returns `{RUN_ID, PROJECT_ID}` from active-run lock table; empty map when idle |
+| `GetAutomationReactor` | `GetAutomation(project=["id"])` | Returns saved pipeline definition (automation.json) |
+| `GetAutomationConfigReactor` | `GetAutomationConfig(project=["id"])` | Returns env var/secret config; masks sensitive values |
+| `GetAutomationRunReactor` | `GetAutomationRun(project=["id"], runId=["id"])` | Returns live run state for FE polling |
+| `ListAutomationRunsReactor` | `ListAutomationRuns(project=["id"])` | Returns run history |
+| `CancelAutomationRunReactor` | `CancelAutomationRun(project=["id"], runId=["id"])` | Cancels an in-progress run |
+| `SaveAutomationReactor` | `SaveAutomation(project=["id"], config=["{}"])` | Persists pipeline definition |
+| `SaveAutomationConfigReactor` | `SaveAutomationConfig(project=["id"], config=["[]"])` | Persists env var config |
+| `RunAutomationNodeReactor` | `RunAutomationNode(project=["id"], nodeId=["id"])` | Single-node test run; result not persisted |
+
+## Node types
+
+Each node type has a corresponding `IAutomationNodeExecutor` in `nodes/`:
+
+| Type | Executor | What it does |
+| --- | --- | --- |
+| `trigger` | (no executor) | Seed node; provides `triggered_at`, `date`, `run_id` scope vars |
+| `database-engine` | `DatabaseEngineNodeExecutor` | Runs SQL via `SqlQuery` pixel |
+| `model-engine` | `ModelEngineNodeExecutor` | LLM ask or embeddings via `IModelEngine` |
+| `vector-engine` | `VectorEngineNodeExecutor` | Search, add, delete, list via `IVectorDatabaseEngine` |
+| `storage-engine` | `StorageEngineNodeExecutor` | File operations via `IStorageEngine` |
+| `function-engine` | `FunctionEngineNodeExecutor` | Function invocation via `IFunctionEngine` |
+| `app` | `AppEngineNodeExecutor` | Arbitrary pixel, optionally scoped to a project |
+| `wait` | `WaitNodeExecutor` | Sleep N seconds; cancel-aware |
+
+## DB tables
+
+| Table | Purpose |
+| --- | --- |
+| `AUTOMATION_ACTIVE_RUN` | PK on `PROJECT_ID` — enforces one concurrent run per project |
+| `AUTOMATION_RUNS` | One row per run: status, timing, node counts |
+| `AUTOMATION_NODE_OUTPUTS` | One row per node per run: status, output, preview, duration |
+
+## Key classes
+
+| Class | Purpose |
+| --- | --- |
+| `AutomationRunEngine` | Orchestrates a full pipeline run end-to-end |
+| `AutomationExecutionUtils` | Shared statics: GSON, scope building, variable resolution, output transforms, preview generation |
+| `AutomationGenerationUtils` | LLM/generation helpers: engine discovery, prompt building, response extraction |
+| `AutomationDatabaseUtility` | All DB reads/writes for runs and node outputs |
+| `PixelExecutionUtils` | Timeout-enforced pixel execution with ThreadStore propagation |
+| `AutomationConstants` | String constants for all keys, statuses, and file names |
+
+---
+
+## Adding a new node type
+
+1. **Create the executor** in `nodes/` implementing `IAutomationNodeExecutor`:
+   - Declare `private static final Logger classLogger = LogManager.getLogger(YourExecutor.class);`
+   - Log execution at `DEBUG` level with node label and key params before dispatching
+   - Use `AutomationExecutionUtils.resolve(value, scope, configMap)` for all `${var}` substitution
+   - Throw `IllegalArgumentException` for missing required config fields
+   - Use `AutomationExecutionUtils.GSON` — do not declare a local `Gson` instance
+
+2. **Register it** in the `EXECUTORS` map on `IAutomationNodeExecutor` (static field on the interface; use `Map.ofEntries` if adding an 11th entry — `Map.of` only supports 10)
+
+3. **Add the type constant** to `AutomationConstants` (e.g. `NODE_TYPE_FOO = "foo-engine"`)
+
+4. **Wire the FE** — add the node type to `automation.types.ts` and `automation.constants.ts` in `SemossWeb`
+
+## Logging rules
+
+Follow the platform standard — SLF4J `{}` placeholders, exception as the last argument:
+
+```java
+// ✅
+classLogger.debug("Foo node \"{}\" executing operation={}", nodeLabel, operation);
+classLogger.error("Foo node \"{}\" failed: {}", nodeLabel, e.getMessage(), e);
+
+// ❌ — never concatenate strings in log calls
+classLogger.error("Foo node " + nodeLabel + " failed: " + e.getMessage());
+```
+
+## Exception conventions
+
+```java
+// Missing/invalid user input — use IllegalArgumentException
+throw new IllegalArgumentException("Foo node \"" + nodeLabel + "\": 'engineId' is required");
+
+// User-facing reactor errors — use SemossPixelException
+throw new SemossPixelException("Project does not exist or user does not have access");
+```
+
+## GSON
+
+Use the shared instance — never declare your own:
+
+```java
+// ✅
+AutomationExecutionUtils.GSON.fromJson(json, AutomationExecutionUtils.MAP_TYPE);
+
+// ❌
+private static final Gson GSON = new GsonBuilder().create();
+```
+
+## Reactor conventions
+
+- Keep reactors thin: parse params, auth-check, delegate, return. Business logic belongs in `AutomationRunEngine`, `AutomationExecutionUtils`, or an executor.
+- Always call `organizeKeys()` before reading `this.keyValue`
+- Use `SecurityProjectUtils.testUserProjectIdForAlias` to resolve alias → UUID before any lookup
+- Add `getReactorDescription()` and `getDescriptionForKey()` to every reactor
+- MCP-destructive reactors (save, trigger, cancel) must override `getMcpToolMetadata()` to `MCPExecution.ASK`
+
+## What not to change
+
+- `AutomationDatabaseUtility` — DB access is intentional; use `setNullableString`, `SelectQueryStruct`, try-with-resources
+- `PixelExecutionUtils` — timeout + ThreadStore propagation; do not bypass
+- `CancelAutomationRunReactor` — dual-signal cancel (DB flag + in-memory); both signals are required for cluster safety
+- `claimActiveRun` — PK-violation is the concurrency guard; do not add a separate lock

src/prerna/reactor/automation/AutomationAskRoomReactor.java

@@ -0,0 +1,202 @@
+/*******************************************************************************
+ * Copyright 2015 Defense Health Agency (DHA)
+ *
+ * If your use of this software does not include any GPLv2 components:
+ * 	Licensed under the Apache License, Version 2.0 (the "License");
+ * 	you may not use this file except in compliance with the License.
+ * 	You may obtain a copy of the License at
+ *
+ * 	  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * 	Unless required by applicable law or agreed to in writing, software
+ * 	distributed under the License is distributed on an "AS IS" BASIS,
+ * 	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * 	See the License for the specific language governing permissions and
+ * 	limitations under the License.
+ * ----------------------------------------------------------------------------
+ * If your use of this software includes any GPLv2 components:
+ * 	This program is free software; you can redistribute it and/or
+ * 	modify it under the terms of the GNU General Public License
+ * 	as published by the Free Software Foundation; either version 2
+ * 	of the License, or (at your option) any later version.
+ *
+ * 	This program is distributed in the hope that it will be useful,
+ * 	but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * 	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * 	GNU General Public License for more details.
+ *******************************************************************************/
+package prerna.reactor.automation;
+
+import prerna.reactor.automation.utils.AutomationGenerationUtils;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.Map;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import prerna.auth.User;
+import prerna.auth.utils.SecurityProjectUtils;
+import prerna.engine.api.IModelEngine;
+import prerna.engine.impl.model.RoomUtils;
+import prerna.reactor.AbstractReactor;
+import prerna.reactor.agent.run.AgentRuntimeManager;
+import prerna.reactor.agent.run.RunAgentRequest;
+import prerna.reactor.agent.run.RunAgentResult;
+import prerna.sablecc2.om.PixelDataType;
+import prerna.sablecc2.om.PixelOperationType;
+import prerna.sablecc2.om.ReactorKeysEnum;
+import prerna.sablecc2.om.nounmeta.NounMetadata;
+import prerna.util.Utility;
+
+/**
+ * Room-aware conversational AI assistant for building automation workflows.
+ * Uses the platform RunAgent harness for server-side history and MCP tool access.
+ * All user engines are registered as MCP tools so the model can query databases,
+ * search vectors, etc. during the design conversation.
+ *
+ * <p>Pixel: {@code AutomationAskRoom(project=["appId"], room=["roomId"], command=["base64text"])}
+ */
+public class AutomationAskRoomReactor extends AbstractReactor {
+
+	private static final Logger classLogger = LogManager.getLogger(AutomationAskRoomReactor.class);
+
+	private static final String ROOM_KEY = "room";
+	private static final String HARNESS_TYPE = "semoss";
+	private static final int MAX_TURNS = 8;
+	private static final long CHAT_TIMEOUT_MS = 180_000L;
+
+	private static final String SYSTEM_PROMPT =
+		"You are an AI assistant that helps users design automation workflows on a no-code platform. "
+		+ "Workflows are linear sequences of steps: database queries, AI model calls, file storage, vector search, or custom functions. "
+		+ "Only manual triggers exist  - do not ask about scheduling.\n\n"
+		+ "You have access to tools. Use them to help the user (for example, query a database to understand its structure "
+		+ "so you can give accurate step descriptions).\n\n"
+		+ "CONVERSATION PHASES:\n\n"
+		+ "Phase 1  - Gather requirements (at most 1-2 short questions, one at a time):\n"
+		+ "- Ask what the automation should do if unclear.\n"
+		+ "- Ask where results should go, or one other essential clarification.\n"
+		+ "- Keep responses under 50 words per question.\n\n"
+		+ "Phase 2  - Plan + build signal (in ONE response, once you have enough info):\n"
+		+ "- Present a concise numbered plain-English plan.\n"
+		+ "- On the very next line after the plan, output the build signal JSON:\n"
+		+ "  {\"action\":\"build\",\"description\":\"<comprehensive 2-4 sentence description of the full workflow>\"}\n"
+		+ "- The description must include all steps in enough detail for an AI to build them.\n"
+		+ "- Do NOT ask 'does this look right?'  - include the build signal in the same response as the plan.\n\n"
+		+ "Phase 3  - If the user requests changes after seeing the plan:\n"
+		+ "- Acknowledge in one short sentence.\n"
+		+ "- Immediately output the revised plan + a new build signal in the same response.\n"
+		+ "- Do NOT narrate what you will change  - just show the revised plan and the signal.\n\n"
+		+ "RULES: Never mention engine IDs, node types, or JSON structure to the user. Be concise.";
+
+	public AutomationAskRoomReactor() {
+		this.keysToGet = new String[] { ReactorKeysEnum.PROJECT.getKey(), ROOM_KEY, ReactorKeysEnum.COMMAND.getKey() };
+		this.keyRequired = new int[] { 1, 0, 1 };
+	}
+
+	@Override
+	public NounMetadata execute() {
+		organizeKeys();
+
+		User user = this.insight.getUser();
+		if (user == null) {
+			throw new IllegalArgumentException("You are not properly logged in.");
+		}
+
+		String projectId = this.keyValue.get(ReactorKeysEnum.PROJECT.getKey());
+		String roomId = this.keyValue.get(ROOM_KEY);
+		String rawCommand = this.keyValue.get(ReactorKeysEnum.COMMAND.getKey());
+
+		projectId = SecurityProjectUtils.testUserProjectIdForAlias(user, projectId);
+		if (!SecurityProjectUtils.userCanViewProject(user, projectId)) {
+			throw new IllegalArgumentException("Project does not exist or user does not have access.");
+		}
+
+		String command = decodeCommand(rawCommand);
+		if (command == null || command.isBlank()) {
+			throw new IllegalArgumentException("command must not be empty.");
+		}
+
+		String engineId = AutomationGenerationUtils.findFirstModelEngine(user);
+		if (engineId == null || engineId.isBlank()) {
+			throw new IllegalArgumentException(
+				"No AI model engine is available. Add a model engine connection to use this feature.");
+		}
+
+		IModelEngine modelEngine = Utility.getModel(engineId);
+		if (modelEngine == null) {
+			throw new IllegalArgumentException("Model engine could not be loaded.");
+		}
+
+		if (roomId == null || roomId.isBlank()) {
+			roomId = "automationchat" + projectId.replace("-", "").substring(0, Math.min(8, projectId.replace("-", "").length()));
+		}
+
+		Map<String, Object> options = AutomationGenerationUtils.buildEngineMcpOptions(user, SYSTEM_PROMPT);
+
+		RoomUtils.createRoomIfNotExists(roomId, this.insight, modelEngine, command, null, options, null, projectId, null);
+
+		RunAgentRequest request = new RunAgentRequest(
+				roomId, command, engineId, HARNESS_TYPE, null,
+				MAX_TURNS, 0, null, null, null, null, this.insight);
+
+		RunAgentResult handle = AgentRuntimeManager.get().run(request);
+		Map<String, Object> result;
+		try {
+			result = AgentRuntimeManager.get().waitForRun(handle.getRunId(), this.insight, CHAT_TIMEOUT_MS);
+		} catch (InterruptedException e) {
+			Thread.currentThread().interrupt();
+			throw new RuntimeException("Chat interrupted.", e);
+		}
+
+		String status = (String) result.get("status");
+		if ("FAILED".equals(status)) {
+			String errMsg = (String) result.get("errorMessage");
+			classLogger.error("AutomationAskRoom run failed: project={} error={}", projectId, (errMsg != null ? errMsg : "unknown error"));
+			throw new RuntimeException("Chat failed: " + (errMsg != null ? errMsg : "unknown error"));
+		}
+
+		String finalText = (String) result.get("finalText");
+		if (finalText == null || finalText.isBlank()) {
+			throw new IllegalStateException("The AI model did not respond. Try again.");
+		}
+
+		classLogger.info("AutomationAskRoom completed: project={}", projectId);
+		return new NounMetadata(finalText.strip(), PixelDataType.CONST_STRING, PixelOperationType.OPERATION);
+	}
+
+	/**
+	 * Decodes a base64-encoded command string. Falls back to the raw value when
+	 * the input is not valid base64 (supports plain-text callers during testing).
+	 */
+	private static String decodeCommand(String raw) {
+		if (raw == null) {
+			return null;
+		}
+		try {
+			return new String(Base64.getDecoder().decode(raw.trim()), StandardCharsets.UTF_8);
+		} catch (Exception e) {
+			return raw;
+		}
+	}
+
+	@Override
+	public String getReactorDescription() {
+		return "Room-aware conversational AI for designing automation workflows. "
+			+ "Uses the platform RunAgent harness with MCP tool access  - the model can query "
+			+ "databases, search vectors, and use other engines during the conversation. "
+			+ "History is managed server-side. "
+			+ "Signals build-readiness via: {\"action\":\"build\",\"description\":\"...\"}";
+	}
+
+	@Override
+	protected String getDescriptionForKey(String key) {
+		return switch (key) {
+			case "project" -> "The project ID the automation belongs to.";
+			case "room" -> "Room ID for this conversation (defaults to automationchat{projectId}).";
+			case "command" -> "The user message, base64-encoded.";
+			default -> super.getDescriptionForKey(key);
+		};
+	}
+}