Automation

[ppatel9703]

Description

Adds backend functionality to our Automation feature - a new AUTOMATION project type that allows users (currently only admins) to build and execute multi-step data pipelines composed of engine calls and control nodes (database, model, vector, storage, function, app, trigger, and wait). Pipelines are authored in the frontend editor and stored as automation.json in the project's app's assets folder. Execution runs asynchronously in a thread pool with per-run heartbeat, cancellation support, and the full run/node history persisting in the Scheduler DB.

Changes Made

New project type

• AUTOMATION added to IProject enum

Core automation package (prerna.reactor.automation)

• AutomationConstants — all column names, table names, status values, and node type strings in one place
• AutomationDatabaseUtility — creates and manages three Scheduler DB tables on startup: AUTOMATION_RUNS, AUTOMATION_NODE_OUTPUTS, AUTOMATION_ACTIVE_RUN; handles heartbeat updates, stale-run interruption on server restart, and cancellation flag
• AutomationExecutionUtils — shared Gson instance, output-transform pipeline (raw, rows-as-objects, first-row, column, JSONPath), output preview truncation, and ${var} template substitution
• AutomationRunEngine — orchestrates a full pipeline run: claims the active-run slot (one run per project at a time),
  seeds the variable scope, walks nodes in order, and runs a background heartbeat thread
• AutomationCancelledException — distinct unchecked exception for clean mid-node cancellation without threading a flag through every call frame

Node executors (nodes/)

• IAutomationNodeExecutor — interface each node type implements
• AutomationNodeExecutors — static registry mapping node type → executor
• Per-type executors: DatabaseEngineNodeExecutor, ModelEngineNodeExecutor, VectorEngineNodeExecutor,
  StorageEngineNodeExecutor, FunctionEngineNodeExecutor, AppEngineNodeExecutor, WaitNodeExecutor

Pixel reactors

• TriggerAutomationReactor — kicks off an async run via a bounded thread pool (2–20 threads); returns {runId}
  immediately
• RunAutomationNodeReactor — executes a single named node in isolation (used for the FE's per-step test/preview)
• GetAutomationReactor — reads automation.json from project storage
• SaveAutomationReactor — writes automation.json to project storage
• GetAutomationConfigReactor — reads automation-config.json (SMSS key-value config) with sensitive values masked
• SaveAutomationConfigReactor — writes automation-config.json with sensitive flag support
• GetAutomationRunReactor — returns a single run's summary + per-node results
• ListAutomationRunsReactor — returns paginated run history for a project
• CancelAutomationRunReactor — sets the cancel flag; the running thread detects it and terminates cleanly
• PixelExecutionUtils — shared helper for executing a Pixel string inside a project insight with a timeout

Startup integration

• SMSSWebWatcher — calls AutomationDatabaseUtility.initialize() on server start to create tables and sweep any
  RUNNING rows left from a previous crash to INTERRUPTED
• SchedulerOwlCreator — registers all new reactors

How to Test

1. Start the server and confirm no errors during AutomationDatabaseUtility.initialize() in the logs
2. In the frontend, create a new Automation app and build a pipeline with at least one Database Engine node and one Model Engine node
3. Click Run — verify the FE receives a runId immediately and polls GetAutomationRun until the run reaches
   SUCCESS or FAILED
4. Open the Scheduler DB and confirm rows appear in AUTOMATION_RUNS and AUTOMATION_NODE_OUTPUTS with correct statuses and durations
5. Click Cancel mid-run — verify the run transitions to CANCELLED (not FAILED) in the DB and the FE reflects this
6. Restart the server while a run is in RUNNING state — verify the row is swept to INTERRUPTED on the next startup

Notes

• The AUTOMATION_ACTIVE_RUN table enforces one concurrent run per project via a PK on PROJECT_ID; attempting to trigger a second run while one is active returns an error
• Output transform (rows-as-objects, JSONPath, etc.) is applied server-side before the preview is stored, keeping the FE display logic thin
• Sensitive config values are never written to AUTOMATION_NODE_OUTPUTS; the *** mask is applied at read time in GetAutomationConfigReactor
• This is Phase 1 — additional nodes and trigger types are planned

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[ppatel9703]

lets make sure to use comments where we can -- and to log following
attaching what i have in my review agent for loggign

Maher's Specific Code Patterns

Logging (His Most Enforced Pattern)

Maher has mass-converted logging across the entire codebase. His rules:

// ❌ NEVER — string concatenation in log statements
classLogger.error("Failed to upload file: " + filePath, e);
classLogger.info("Sync completed for: " + storagePath);

// ✅ ALWAYS — SLF4J {} placeholder notation
classLogger.error("Failed to upload file: {}", filePath, e);
classLogger.info("Sync completed for: {}", storagePath);

Critical detail: When logging exceptions, the exception object goes as the LAST argument after all placeholders:

// ✅ Correct — exception is last arg, not in placeholder
classLogger.error("Error processing {} for user {}", engineId, userId, e);

// ❌ Wrong — exception consumed by placeholder
classLogger.error("Error processing {} for user {} with error {}", engineId, userId, e);

No System.out.println — ever. Use classLogger.

Conditional log messages — for ternary log statements, Maher prefers if/else when the messages are substantially different:

// ✅ Maher's preferred style for divergent messages
if (uploadedFiles.isEmpty()) {
    classLogger.info("No files were uploaded.");
} else {
    classLogger.info("Successfully uploaded files: {}", uploadedFiles);
}

// ✅ Acceptable for simple variant messages
classLogger.info(found ? "Sync completed for: {}" : "No files found for: {}", storagePath);

Exception Handling

// ✅ Throw SemossPixelException for user-facing errors in reactors
throw new SemossPixelException("Engine does not exist or user does not have access");

// ✅ IllegalArgumentException for validation errors
throw new IllegalArgumentException("Must pass in the project id");

// ✅ RuntimeException wrapping cause for infrastructure errors
throw new RuntimeException("Invalid S3 endpoint URI: " + this.endpoint, e);

// ❌ Don't throw errors for recoverable situations — return gracefully
// Instead of: throw new SemossPixelException("No MCP tools found");
// Do: return new NounMetadata(new ArrayList<>(), PixelDataType.MAP);

Key pattern from PR #2319: Instead of throwing an error, return no tools if engine MCP is not enabled. Maher prefers graceful degradation over hard failures when the situation is recoverable.

[ppatel9703]

similarly lets add proper java docs, will send you my agent file to run a local review

[ppatel9703]

I know this contradicts what i said on the call -- but i think for databases we should use the reactor or the some util class they do for SqlQueryReactor

[ppatel9703]

What is the main difference between this and getautomation

[ppatel9703]

not sure if this file or list runs
but we need to scope the timezone to the users location on return
might just be a bug on my local

[ppatel9703]

do we still need edges, graphs and stuff since we moved to form

[ppatel9703]

we should make a utiility for this file -- its really large otherwise
we should try to make the reactors as lightweight as we can

[ppatel9703]

Lets also add a readme to the folders and a agent.md (similar to the one i sent in the chat)