feat: flyway style migration

[ppatel9703]

Description

Adds Flyway-style versioned schema migrations for RDBMS-backed database engines, gated per-engine behind an ENABLE_MIGRATIONS smss flag. Migration files (V<version>__<description>.sql) live in the engine's own assets/.migrations folder, are tracked via a history table inside the target engine's own database, and are run automatically on engine open() as well as on-demand when a new migration is authored through the UI. This is the backend counterpart to the Migrations tab UI added in SEMOSS/semoss-ui#3500 — see How It Ties Together below for the full request lifecycle across both PRs.

Changes Made

Gating & trigger — Constants.ENABLE_MIGRATIONS is a plain smss property read per-engine. In RDBMSNativeEngine.open(), after the existing connection-establishment step (which only logs on SQLException and never throws), a new block runs runPendingMigrations() when engineConnected && ENABLE_MIGRATIONS=true. Unlike the connection-failure path, a migration failure is not caught here — it propagates out of open() on purpose, since Utility.loadEngine() never registers an engine whose open() throws. A bad migration leaves the engine fully unusable rather than usable with a half-migrated schema.

File discovery (MigrationFile, MigrationFileUtils) — Files must match ^V(\d+(?:\.\d+)*)__(.+)\.sql$ (case-insensitive) and live under assets/.migrations (resolved via EngineUtility.getSpecificEngineAssetsFolder). Non-matching files are logged as a warning and silently skipped, never treated as an error. Versions sort numerically segment-by-segment ("2" < "2.1" < "10", not lexicographic), and each file's checksum is a SHA-256 of its raw content. Statement splitting is intentionally naive (line-based, split on ;-terminated lines) — does not handle semicolons inside string literals or stored procedures, called out in-code as out of scope for v1.

History (MigrationHistoryRecord, MigrationHistoryUtils) — Persisted in a SEMOSS_SCHEMA_HISTORY table created inside the engine's own database (not a central SEMOSS table): VERSION, SCRIPTNAME, CHECKSUM, APPLIEDBY, APPLIEDON, EXECUTIONTIMEMS, SUCCESS, DESCRIPTION (the description column doubles as the failure-reason text on failed rows). Reads use raw JDBC rather than SelectQueryStruct, because the history table only becomes an OWL-registered concept after the first migration's OWL sync — querying it through the query-struct layer would fail during that bootstrap window. Every attempt inserts a new row (never updates), so retries are fully auditable.

Status (MigrationStatus, MigrationStatusUtils) — Merges the on-disk file scan with the latest history record per version (by max appliedOn, since retries produce multiple rows) into one of five states: PENDING (file, no history), SUCCESS, FAILED (last attempt failed), OUTDATED (succeeded previously but on-disk checksum no longer matches the recorded one), MISSING (history exists but the file is gone).

Locking (SchemaMigrationLock) — Dialect-dispatched, DB-level, not file- or JVM-local: Postgres uses a real session-level advisory lock (pg_try_advisory_lock/pg_advisory_unlock) keyed by a hash of the engine ID; MySQL/H2 use a SEMOSS_SCHEMA_LOCK table with a PRIMARY KEY on ENGINEID so acquisition is race-free via constraint violation. A lock row older than 600s is treated as abandoned and stolen. Waits retry every 200ms up to a 30s timeout, then throw SchemaMigrationLockTimeoutException.

Runner (SchemaMigrationRunner) — Orchestration: acquire lock → ensure folder/history table exist → scan files → for each in version order: skip already-applied files after verifying checksum is unchanged (throws on mismatch), reject if version is lower than the highest already-applied version (out-of-order), reject re-running a version that previously failed with the same checksum (must fix the content first) → run the migration in its own transaction (own connection, setAutoCommit(false), statements via PreparedStatement, history row inserted on the same transaction so SQL + history commit or roll back together) → sync OWL for that file immediately (not batched at the end, so a later failure doesn't leave earlier schema changes unsynced by OWL). On failure, the transaction rolls back and the failure is recorded on a fresh connection (so the failure record survives the rollback). On a lock-timeout, the runner re-checks history first — if another node already finished the migration, open() proceeds normally instead of failing.

OWL sync (RdbmsMigrationOwlSyncUtils) — Because migrations execute raw SQL directly over JDBC (bypassing SEMOSS's normal upload/edit reactors), the OWL metamodel that SelectQueryStruct relies on would otherwise go stale. This reuses the same discovery/diff logic as the Metadata tab's "Sync" button (RDBMSEngineCreationHelper for JDBC introspection, UploadUtilities for the OWL side), called as plain Java since there's no Insight/user context at open() time. SEMOSS_SCHEMA_LOCK is excluded from the diff (transient plumbing); SEMOSS_SCHEMA_HISTORY is deliberately included as an engine-managed table. Adds new concepts/props/FKs; known limitation: removed foreign keys are not currently pruned from OWL (add-only).

Reactors — GetEngineMigrationsEnabledReactor (input: engine; requires view permission; returns the raw smss boolean). ListEngineMigrationsReactor (input: engine; requires view permission; returns MigrationStatusUtils.getStatus() as a vector of maps). SaveEngineMigrationReactor (inputs: engine, sql, description; requires edit permission): acquires the lock, computes the next version as highestMajorSegment + 1 (dotted sub-versions like 2.1 are collapsed — next after 2.1 is 3, not 2.2), sanitizes the description ([^a-zA-Z0-9-_] → _), writes the file, releases the lock, then immediately runs it through the same SchemaMigrationRunner pipeline used by open(). On failure it distinguishes: if the newly-saved version itself failed, the just-written file is deleted so it doesn't permanently block the chain, and the recorded error is returned; if an earlier pending version failed instead, the new file is kept (queued) and a "queued behind an earlier failure" message is returned. Declares SMSS_MCP_EXECUTION=ASK so it can never be agent-auto-triggered, since it creates and immediately executes arbitrary DDL/DML.

FileSystemUtil.java — Extended the hidden-asset checks (isHiddenName/isHiddenAsset/isWithinHiddenAsset) to also treat .migrations as hidden, alongside .git/.admin, so engine file-explorer/search endpoints never expose or traverse into it.

Tests — SchemaMigrationRunnerUnitTests covers rejectIfPreviouslyFailedUnchanged (throws on same version+checksum already failed; no-op when content changed, no history exists, only a success record exists, or the failure was for a different version). SaveEngineMigrationReactorUnitTests covers handleRunFailure: deletes the file and surfaces the error when the just-saved version failed; keeps the file and returns a "queued" message when an earlier version failed instead; falls back to the caught exception's message when no status row exists at all.

How It Ties Together

1. UI loads an engine page → calls GetEngineMigrationsEnabled → Migrations tab renders only if this engine's smss has ENABLE_MIGRATIONS=true.
2. Tab renders → UI calls ListEngineMigrations → this PR's MigrationStatusUtils.getStatus() reconciles assets/.migrations on disk against SEMOSS_SCHEMA_HISTORY in the engine's own DB and returns one row per version with its state.
3. User submits a new migration (description + SQL) → UI calls SaveEngineMigration → this PR writes the V<n>__desc.sql file, runs it immediately through SchemaMigrationRunner, syncs OWL, and returns the outcome — UI then re-calls ListEngineMigrations to refresh the table.
4. Independently of the UI, every time the engine itself is opened (server restart, engine reconnect), RDBMSNativeEngine.open() runs any migration files that were added out-of-band (e.g. deployed alongside a release) so environments stay in sync without requiring someone to click through the UI.
5. The frontend's MigrationState type (PENDING/SUCCESS/FAILED/MISSING/OUTDATED) is a direct mirror of this PR's MigrationStatus states — both sides comment the contract explicitly to keep them from drifting.

How to Test

1. Set ENABLE_MIGRATIONS=true on a DATABASE engine's smss properties.
2. Via SaveEngineMigration (or the paired UI), create and run a new versioned SQL migration; confirm the file lands in assets/.migrations as V<version>__<description>.sql and a history row is written in SEMOSS_SCHEMA_HISTORY.
3. Restart/reopen the engine and confirm already-applied migrations are not re-run, while any new pending file is applied on open().
4. Call ListEngineMigrations and confirm status reflects reality — including forcing an OUTDATED state by editing an already-applied file's content, and a MISSING state by deleting a file that has history.
5. Submit a migration with intentionally invalid SQL and confirm it's recorded FAILED, the file is deleted, and a subsequent valid migration at the same version number can be resubmitted.
6. Attempt two concurrent SaveEngineMigration/open() calls against the same engine and confirm SchemaMigrationLock serializes them (no double-run, no duplicate version numbers).
7. Confirm OWL metadata reflects new columns/tables/FKs after a migration (e.g. via the Metadata tab) without needing a manual "Sync".
8. Run SchemaMigrationRunnerUnitTests and SaveEngineMigrationReactorUnitTests.

Notes

• Depends on / pairs with SEMOSS/semoss-ui#3500, which surfaces this via the Migrations tab.
• Opt-in per engine — migrations never run unless ENABLE_MIGRATIONS is explicitly set.
• Known follow-ups: FK removals aren't pruned from OWL (add-only sync); SaveEngineMigration's next-version logic collapses dotted sub-versions to the next integer major version rather than incrementing the last segment.

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.