Merge codegen-kotlin Phase N — close persistence-conformance gaps (11/12)

dmealing · dmealing · commit d22c153cf215 · 2026-05-26T07:45:34.000-04:00
QueryScenarioRunner gained the missing operator vocabulary:
- isNull / isNotNull (via IsNullOp/IsNotNullOp constructors — not top-level fns)
- ne (neq), like, gt/gte/lt/lte (with Comparable&lt;T&gt; cast inside SqlExpressionBuilder)
- in (inList)
- top-level 'and: [...]' filter combinator

5 query scenarios newly green: filter-by-enum, filter-is-null,
filter-like-and-ne, filter-range-and, projection-aggregate (last one
needed a new ProgramStatView reference table + correlated-subquery
CREATE VIEW emitted at runner setup).

1 migration scenario newly green: add-nullable-column via Exposed's
SchemaUtils.addMissingColumnsStatements(ProgramV2Table).

Sole remaining deferral: drop-table-blocked-without-allow — Exposed's
diff helpers only ADD, never DROP; the "blocked destructive change"
semantic is part of Java OMDB's SchemaMigrationEngine + AllowOptions
with NO substrate analogue in Exposed. Closing it would require a
Kotlin migration façade on top of Exposed (~mini migration engine).

Lexical SQL divergence between Exposed and Java/TS/C# emitted ALTER
syntax handled via sqlSignature() shape-normalizer (case-insensitive,
quote/COLUMN-keyword/whitespace tolerant) so structural conformance
is verified without lexical coupling.

11 of 12 persistence-conformance scenarios green via Exposed against
Testcontainers Postgres.
diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/MigrationScenarioConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/MigrationScenarioConformanceTest.kt
@@ -1,6 +1,8 @@
 package com.metaobjects.integration.kotlin
 
 import com.metaobjects.integration.kotlin.tables.ProgramTable
+import com.metaobjects.integration.kotlin.tables.ProgramV1Table
+import com.metaobjects.integration.kotlin.tables.ProgramV2Table
 import com.metaobjects.integration.kotlin.tables.WeekTable
 import org.jetbrains.exposed.sql.Database
 import org.jetbrains.exposed.sql.SchemaUtils
@@ -14,12 +16,26 @@ import kotlin.test.assertTrue
  * Migration scenario coverage for the Kotlin/Exposed substrate. Exposed's
  * `SchemaUtils.create(*tables)` is the analogue of the sibling Java port's
  * "emit + apply CREATE DDL against the live connection" loop — both produce
- * the canonical base-table set from an empty Postgres.
+ * the canonical base-table set from an empty Postgres. For additive column
+ * migrations, `SchemaUtils.addMissingColumnsStatements(*tables)` is the
+ * analogue of `SchemaMigrationEngine.emit(...)` producing an ALTER TABLE
+ * ADD COLUMN statement.
  *
- * We exercise the `bootstrap-canonical-from-empty` scenario specifically: it
- * is the simplest migration in the corpus and its post-condition (the
- * `apply-up-then-query` block) lists the expected base-table set, which is
- * the natural assertion for the Exposed substrate.
+ * Scenarios:
+ *  - [bootstrap_canonical_from_empty]: SchemaUtils.create(...) the canonical
+ *    base-table set. Asserts against the apply-up-then-query post-condition.
+ *  - [add_nullable_column]: bootstrap v1 (id + title only), seed two rows,
+ *    invoke addMissingColumnsStatements with v2 → assert it surfaces
+ *    `ALTER TABLE ... ADD COLUMN "subtitle" VARCHAR(200)` (Exposed's quoting
+ *    matches the corpus `up-contains` expectation), apply it, then query.
+ *
+ * Deferred:
+ *  - drop-table-blocked-without-allow: needs a destructive-change-blocking
+ *    pipeline. Exposed has no analogue — its diff helpers
+ *    (`addMissingColumnsStatements` / `statementsRequiredToActualizeScheme`)
+ *    only ADD, never DROP. The "blocked" semantic is part of OMDB's
+ *    SchemaMigrationEngine + AllowOptions, not the substrate. Re-evaluate
+ *    only if a Kotlin-side migration façade is built on top of Exposed.
  *
  * NOT registered in the parent reactor — run via:
  *   mvn -f server/java/integration-tests-kotlin/pom.xml test
@@ -30,8 +46,8 @@ internal class MigrationScenarioConformanceTest {
     fun `bootstrap canonical from empty creates the expected base tables`() {
         val corpus = ScenarioLoader.findCorpusRoot()
         val scenarios = ScenarioLoader.loadMigrations(corpus.resolve("migrations"))
-        val bootstrap = scenarios.singleOrNull { it.name == BOOTSTRAP_NAME }
-            ?: error("Expected scenario '$BOOTSTRAP_NAME' in corpus")
+        val bootstrap = scenarios.singleOrNull { it.name == "bootstrap-canonical-from-empty" }
+            ?: error("Expected scenario 'bootstrap-canonical-from-empty' in corpus")
 
         PostgresContainer().use { pg ->
             val db = Database.connect(pg.jdbcUrl, user = pg.username, password = pg.password)
@@ -76,6 +92,109 @@ internal class MigrationScenarioConformanceTest {
         }
     }
 
+    @Test
+    fun `add nullable column emits ALTER TABLE ADD COLUMN and preserves existing rows`() {
+        val corpus = ScenarioLoader.findCorpusRoot()
+        val scenarios = ScenarioLoader.loadMigrations(corpus.resolve("migrations"))
+        val scenario = scenarios.singleOrNull { it.name == "add-nullable-column" }
+            ?: error("Expected scenario 'add-nullable-column' in corpus")
+
+        PostgresContainer().use { pg ->
+            val db = Database.connect(pg.jdbcUrl, user = pg.username, password = pg.password)
+
+            // 1. Bootstrap the v1 schema — `id` + `title` only.
+            transaction(db) {
+                SchemaUtils.create(ProgramV1Table)
+            }
+
+            // 2. Seed the v1 rows from the scenario's seed-data SQL.
+            scenario.seedData?.takeIf { it.isNotBlank() }?.let { sql ->
+                DriverManager.getConnection(pg.jdbcUrl, pg.username, pg.password).use { c ->
+                    c.createStatement().use { it.execute(sql) }
+                }
+            }
+
+            // 3. Generate ALTER statements from the v2 shape vs the live schema.
+            //    `addMissingColumnsStatements` is Exposed's diff-and-emit helper for
+            //    additive column changes — the substrate-level analogue of the
+            //    sibling Java port's SchemaMigrationEngine.emit() for ADD COLUMN.
+            val alterStatements = transaction(db) {
+                SchemaUtils.addMissingColumnsStatements(ProgramV2Table)
+            }
+
+            // 4. Substrate-aware up-contains assertion.
+            //    The corpus's up-contains needles are Java/TS/C#-emitter-shaped
+            //    (`ALTER TABLE "programs" ADD COLUMN "subtitle" VARCHAR(200)`).
+            //    Exposed's emitter produces semantically-equivalent SQL with
+            //    different surface form: no identifier quoting on bare lowercase
+            //    tables, no `COLUMN` keyword, trailing `NULL` for nullable.
+            //    We normalize both sides to a shape-only signature and assert
+            //    the structural pieces match — captures the real conformance
+            //    contract (ADD `subtitle` of type VARCHAR(200) on `programs`)
+            //    while tolerating the substrate's lexical divergence.
+            val joined = alterStatements.joinToString("\n")
+            for (needle in scenario.expect.upContains) {
+                val sig = sqlSignature(needle)
+                val joinedSig = sqlSignature(joined)
+                assertTrue(
+                    joinedSig.contains(sig),
+                    "expected ALTER statements to contain shape-equivalent of '$needle'\n" +
+                        "  needle signature : '$sig'\n" +
+                        "  emitted SQL      : $joined\n" +
+                        "  emitted signature: '$joinedSig'"
+                )
+            }
+
+            // 5. Apply the emitted statements then run the post-condition query.
+            transaction(db) {
+                for (stmt in alterStatements) {
+                    exec(stmt)
+                }
+            }
+
+            val auq = scenario.expect.applyUpThenQuery
+                ?: error("add-nullable-column scenario must declare apply-up-then-query")
+            val actualRows = DriverManager.getConnection(pg.jdbcUrl, pg.username, pg.password).use { c ->
+                c.createStatement().use { stmt ->
+                    stmt.executeQuery(auq.sql).use { rs ->
+                        val out = mutableListOf<Map<String, Any?>>()
+                        val meta = rs.metaData
+                        while (rs.next()) {
+                            val row = LinkedHashMap<String, Any?>(meta.columnCount)
+                            for (i in 1..meta.columnCount) row[meta.getColumnLabel(i)] = rs.getObject(i)
+                            out.add(row)
+                        }
+                        out
+                    }
+                }
+            }
+            val expectedJson = Normalization.canonicalRowsJson(auq.rows)
+            val actualJson = Normalization.canonicalRowsJson(actualRows)
+            assertEquals(expectedJson, actualJson, "add-nullable-column post-query mismatch")
+        }
+    }
+
+    /**
+     * Normalize a SQL fragment to a shape-only signature so different emitters
+     * (Java/TS/C# vs Exposed) can be compared structurally without coupling to
+     * each substrate's lexical choices (identifier quoting, optional COLUMN
+     * keyword, trailing NULL/NOT NULL on additive nullable columns).
+     *
+     * Strips:
+     *  - all double quotes (`"id"` → `id`)
+     *  - the `COLUMN` keyword between `ADD` and the column name
+     *  - trailing `NULL` (additive nullable column — implicit in both shapes)
+     *  - all whitespace runs collapsed to a single space
+     *  - lower-cases everything for case-insensitive keyword matching
+     */
+    private fun sqlSignature(sql: String): String =
+        sql.replace("\"", "")
+            .replace(Regex("\\bCOLUMN\\b", RegexOption.IGNORE_CASE), "")
+            .replace(Regex("\\bNULL\\b\\s*$", RegexOption.IGNORE_CASE), "")
+            .replace(Regex("\\s+"), " ")
+            .trim()
+            .lowercase()
+
     private fun assertColumnsExist(pg: PostgresContainer, table: String, expectedColumns: List<String>) {
         val sql = """
             SELECT column_name FROM information_schema.columns
@@ -98,8 +217,4 @@ internal class MigrationScenarioConformanceTest {
             )
         }
     }
-
-    companion object {
-        private const val BOOTSTRAP_NAME = "bootstrap-canonical-from-empty"
-    }
 }
diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioConformanceTest.kt
@@ -35,30 +35,29 @@ internal class QueryScenarioConformanceTest {
 
     companion object {
         /**
-         * Curated Day-1 subset — these scenarios only use op=count|get|list with at
-         * most `eq` filters and ascending sorts, all of which the Exposed runner
-         * handles today. Listed explicitly (not derived) so a new scenario file
-         * lands as 'skipped' rather than silently expanding the suite.
+         * Included scenarios — these run end-to-end against the Exposed substrate.
+         * Listed explicitly (not derived) so a new scenario file lands as 'skipped'
+         * rather than silently expanding the suite.
          */
         private val INCLUDED_SCENARIOS = setOf(
             "count",
             "get-by-id",
             "list-empty-table",
             "list-programs-sorted",
+            "filter-by-enum",
+            "filter-is-null",
+            "filter-like-and-ne",
+            "filter-range-and",
+            "projection-aggregate",
         )
 
         /**
-         * Scenarios deliberately left out of the Day-1 subset, with a one-line
-         * reason. Kept here so the deferral surface is visible in code review.
+         * Scenarios deliberately left out, with a concrete reason describing
+         * what would unblock each. Kept visible so the deferral surface is
+         * obvious in code review.
          */
         @Suppress("unused")
-        private val DEFERRED_SCENARIOS = mapOf(
-            "filter-by-enum" to "needs eq on string enum — works but corpus also exercises non-eq variants in sibling scenarios; bundled",
-            "filter-is-null" to "needs isNull operator — not yet wired in QueryScenarioRunner.applyFilter",
-            "filter-like-and-ne" to "needs like/ne operators — not yet wired",
-            "filter-range-and" to "needs gt/lt/range operators — not yet wired",
-            "projection-aggregate" to "needs view tables + aggregate projection — out of subset scope",
-        )
+        private val DEFERRED_SCENARIOS = emptyMap<String, String>()
 
         @JvmStatic
         fun scenarios(): Stream<Arguments> {
diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioRunner.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioRunner.kt
@@ -2,6 +2,7 @@ package com.metaobjects.integration.kotlin
 
 import com.metaobjects.integration.kotlin.Scenarios.QueryScenario
 import com.metaobjects.integration.kotlin.Scenarios.QuerySpec
+import com.metaobjects.integration.kotlin.tables.ProgramStatView
 import com.metaobjects.integration.kotlin.tables.ProgramTable
 import com.metaobjects.integration.kotlin.tables.WeekTable
 import org.jetbrains.exposed.sql.AndOp
@@ -13,6 +14,8 @@ import org.jetbrains.exposed.sql.ResultRow
 import org.jetbrains.exposed.sql.SchemaUtils
 import org.jetbrains.exposed.sql.SortOrder
 import org.jetbrains.exposed.sql.SqlExpressionBuilder
+import org.jetbrains.exposed.sql.IsNotNullOp
+import org.jetbrains.exposed.sql.IsNullOp
 import org.jetbrains.exposed.sql.Table
 import org.jetbrains.exposed.sql.selectAll
 import org.jetbrains.exposed.sql.transactions.transaction
@@ -58,6 +61,15 @@ object QueryScenarioRunner {
             }
         }
 
+        // 2b. Projection views — only created when a scenario actually touches
+        // the relevant projection entity. Cheap and explicit; avoids running
+        // view DDL on scenarios that don't need it.
+        if (scenario.queries.any { it.entity == "ProgramStat" }) {
+            DriverManager.getConnection(pg.jdbcUrl, pg.username, pg.password).use { c ->
+                c.createStatement().use { it.execute(ProgramStatView.VIEW_DDL) }
+            }
+        }
+
         // 3. Run queries; each gets its own Exposed transaction.
         for (spec in scenario.queries) {
             val actual = transaction(db) { dispatch(spec) }
@@ -97,6 +109,7 @@ object QueryScenarioRunner {
     private fun tableFor(entity: String): Table = when (entity) {
         "Program" -> ProgramTable
         "Week" -> WeekTable
+        "ProgramStat" -> ProgramStatView
         else -> error("No Exposed Table registered for entity '$entity' — extend QueryScenarioRunner.tableFor")
     }
 
@@ -115,29 +128,81 @@ object QueryScenarioRunner {
     }
 
     /**
-     * Translate a YAML filter block into Exposed `Op<Boolean>`. Only the
-     * operators surfaced by the curated query-scenario subset are
-     * implemented (eq); broader operator support belongs in a follow-up
-     * once more scenarios land.
+     * Translate a YAML filter block into Exposed `Op<Boolean>`. Supports the
+     * cross-language filter vocabulary: eq, ne, gt, gte, lt, lte, in, like,
+     * isNull. Also supports the top-level `and: [...]` combinator the corpus
+     * uses for two-op-on-one-field range queries (filter-range-and).
      */
     @Suppress("UNCHECKED_CAST")
     private fun applyFilter(q: Query, table: Table, filter: Map<String, Any?>?) {
         if (filter.isNullOrEmpty()) return
+        val combined = buildFilterOp(table, filter) ?: return
+        q.adjustWhere { combined }
+    }
+
+    /**
+     * Recursively build an `Op<Boolean>` from a filter map. A filter map is
+     * either:
+     *  - a single-entry map `{ and: [<filter>, <filter>, ...] }` (combinator),
+     *  - or a per-field map `{ fieldName: { op: value, ... }, ... }` ANDed together.
+     */
+    @Suppress("UNCHECKED_CAST")
+    private fun buildFilterOp(table: Table, filter: Map<String, Any?>): Op<Boolean>? {
+        // Combinator: `and: [...]` — recurse on each child filter map, AND together.
+        if (filter.size == 1 && filter.containsKey("and")) {
+            val list = filter["and"] as? List<Map<String, Any?>>
+                ?: error("`and` value must be a list of filter maps; got: ${filter["and"]}")
+            val childOps = list.mapNotNull { buildFilterOp(table, it) }
+            return combine(childOps)
+        }
         val ops = mutableListOf<Op<Boolean>>()
         for ((field, opsRaw) in filter) {
             val col = columnFor(table, field)
             val opMap = opsRaw as? Map<String, Any?>
                 ?: error("filter for '$field' must be an op-map (e.g. { eq: ... }); got: $opsRaw")
             for ((op, raw) in opMap) {
-                val coerced = coerce(raw, col)
-                ops += when (op) {
-                    "eq" -> buildEq(col, coerced)
-                    else -> error("Unsupported filter op '$op' for ${col.name} — extend QueryScenarioRunner.applyFilter")
-                }
+                ops += buildOp(col, op, raw)
             }
         }
-        val combined = combine(ops) ?: return
-        q.adjustWhere { combined }
+        return combine(ops)
+    }
+
+    /**
+     * Build a single Exposed `Op<Boolean>` from `(column, op-name, raw-value)`.
+     * `isNull` is value-shaped: `{ isNull: true }` → IS NULL, `{ isNull: false }` → IS NOT NULL.
+     */
+    @Suppress("UNCHECKED_CAST")
+    private fun buildOp(col: Column<*>, op: String, raw: Any?): Op<Boolean> {
+        val any = col as Column<Any?>
+        // Comparable cast — gt/gte/lt/lte require `T: Comparable<T>`; YAML scalars
+        // (Long, String, Instant) all satisfy that at runtime, but the static type
+        // system needs the explicit shape. UNCHECKED_CAST suppression covers it.
+        val cmp = col as Column<Comparable<Any>>
+        return when (op) {
+            "eq" -> buildEq(col, coerce(raw, col))
+            "ne" -> with(SqlExpressionBuilder) { any neq coerce(raw, col) }
+            "gt" -> with(SqlExpressionBuilder) { cmp greater (coerce(raw, col) as Comparable<Any>) }
+            "gte" -> with(SqlExpressionBuilder) { cmp greaterEq (coerce(raw, col) as Comparable<Any>) }
+            "lt" -> with(SqlExpressionBuilder) { cmp less (coerce(raw, col) as Comparable<Any>) }
+            "lte" -> with(SqlExpressionBuilder) { cmp lessEq (coerce(raw, col) as Comparable<Any>) }
+            "like" -> {
+                val pattern = (raw as? String)
+                    ?: error("`like` value must be a string pattern; got: $raw")
+                with(SqlExpressionBuilder) { (col as Column<String>) like pattern }
+            }
+            "in" -> {
+                val list = (raw as? List<Any?>)
+                    ?: error("`in` value must be a list; got: $raw")
+                val coerced = list.map { coerce(it, col) }
+                with(SqlExpressionBuilder) { any inList coerced }
+            }
+            "isNull" -> {
+                val flag = raw as? Boolean
+                    ?: error("`isNull` value must be a boolean; got: $raw")
+                if (flag) IsNullOp(col) else IsNotNullOp(col)
+            }
+            else -> error("Unsupported filter op '$op' for ${col.name} — extend QueryScenarioRunner.buildOp")
+        }
     }
 
     /**
diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramStatView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramStatView.kt
@@ -0,0 +1,41 @@
+package com.metaobjects.integration.kotlin.tables
+
+import org.jetbrains.exposed.sql.Table
+
+/**
+ * Hand-written reference Exposed Table mapping the `ProgramStat` projection
+ * from `fixtures/persistence-conformance/canonical/meta.fitness.json`.
+ *
+ * Backed by the Postgres VIEW `v_program_stat` (NOT a base table) — see
+ * [VIEW_DDL] for the correlated-subquery aggregate body the
+ * [com.metaobjects.integration.kotlin.QueryScenarioRunner] issues alongside
+ * `SchemaUtils.create(...)` for scenarios touching this projection.
+ *
+ * Exposed has no first-class "view" Table — using `Table("v_program_stat")`
+ * is the conventional Exposed pattern for read-only SELECTs over a view:
+ * `selectAll()` issues `SELECT * FROM v_program_stat`, exactly what the
+ * projection-aggregate scenario expects. We never call `SchemaUtils.create`
+ * on this object — the view is created via raw SQL.
+ */
+object ProgramStatView : Table("v_program_stat") {
+    val programId = long("programId")
+    val weekCount = long("weekCount")
+
+    override val primaryKey = PrimaryKey(programId)
+
+    /**
+     * Correlated-subquery aggregate body, matching the metadata's
+     * `origin.aggregate{@agg: count, @of: Week.id, @via: Program.weeks}`
+     * shape and the canonical view-DDL pattern documented in
+     * `docs/superpowers/specs/2026-05-22-fr-003-*` for cross-language ports.
+     *
+     * Identifier quoting matches the seed DDL: bare lowercase for table
+     * names, double-quoted camelCase for column names — same convention
+     * `SchemaUtils.create(ProgramTable, WeekTable)` produces.
+     */
+    const val VIEW_DDL =
+        """CREATE OR REPLACE VIEW "v_program_stat" AS
+           SELECT p."id" AS "programId",
+                  (SELECT COUNT(w."id") FROM "weeks" w WHERE w."programId" = p."id") AS "weekCount"
+             FROM "programs" p"""
+}
diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramV1Table.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramV1Table.kt
diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramV2Table.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramV2Table.kt
