|
| 1 | +# R6 Plan 2c — Python introspection parity (cross-port) |
| 2 | + |
| 3 | +_Date: 2026-05-30. Status: **Design — approved, pending spec review.**_ |
| 4 | + |
| 5 | +> Part of R6 (RDB fidelity). Sequenced **first** in Plan 2 (foundation), before |
| 6 | +> **Plan 2a (`field.uuid`)** and **Plan 2b (`@dbColumnType`)**, whose introspection |
| 7 | +> round-trips become verifiable in Python once this lands. |
| 8 | +> |
| 9 | +> Principle: **all five ports the same implementation/capability.** Python is the only |
| 10 | +> port whose migrate engine cannot read a live database. Governed by |
| 11 | +> [ADR-0007](../../../spec/decisions/ADR-0007-source-v2-paradigm.md) (RDB persistence/migrate |
| 12 | +> paradigm) and [ADR-0013](../../../spec/decisions/ADR-0013-logical-field-types-vs-physical-column-attributes.md) |
| 13 | +> §33 ("a physical attribute round-trips from the live schema via introspection"). |
| 14 | +> Mirrors existing reference implementations per the project rule *study reference |
| 15 | +> implementations; don't re-derive from spec.* |
| 16 | +
|
| 17 | +## Context |
| 18 | + |
| 19 | +The migrate engine converges a database by **diffing two schemas**: the **expected** schema |
| 20 | +(built from metadata) against the **actual** schema (read back from the live database), emitting |
| 21 | +only the `CREATE`/`ALTER` needed. Reading the actual schema back — querying the DB catalog and |
| 22 | +reconstructing it as the port's `SqlType` model — is **introspection**. It is what enables |
| 23 | +*incremental* migration against an existing database, not just bootstrap-from-empty. |
| 24 | + |
| 25 | +### The gap (audit, 2026-05-30) |
| 26 | + |
| 27 | +Four ports have an introspection layer; **Python has none**: |
| 28 | + |
| 29 | +| Port | Introspection module | |
| 30 | +|---|---| |
| 31 | +| TS | `migrate-ts/src/introspect/postgres.ts` (+ sqlite, d1) | |
| 32 | +| C# | `MetaObjects.Codegen/Migrate/PostgresIntrospect.cs` (+ CLI `NpgsqlIntrospector.cs`) | |
| 33 | +| Java | `omdb/.../migrate/SchemaIntrospector.java` (+ `JdbcSqlTypes.java`) | |
| 34 | +| **Python** | **none** — `migrate/` has `expected_schema.py`, `diff.py`, `postgres_emit.py`, `sql_type.py`, `types.py`, but no `introspect*` | |
| 35 | + |
| 36 | +Python's `diff(expected, actual)` already accepts a `SchemaSnapshot` for `actual`, but `actual` is |
| 37 | +always **empty** today (`diff.py`: "Bootstrap path (empty actual) only"). So Python can create a |
| 38 | +schema from empty but cannot read an existing database and converge it — the one port that can't. |
| 39 | +This also makes the ADR-0013 physical-attribute litmus (uuid/jsonb/timestamptz round-trip via |
| 40 | +introspection) unverifiable in Python, blocking full parity for Plans 2a/2b. |
| 41 | + |
| 42 | +## Decision / Design |
| 43 | + |
| 44 | +Build a Python introspection layer that mirrors the existing reference implementations and feeds |
| 45 | +the **already-present** `diff` unchanged. |
| 46 | + |
| 47 | +### 1. New module: `server/python/src/metaobjects/migrate/introspect_postgres.py` |
| 48 | + |
| 49 | +A single public function mirroring Java's `SchemaIntrospector.introspect(conn, schema)`: |
| 50 | + |
| 51 | +```python |
| 52 | +def introspect(conn, schema: str = "public") -> SchemaSnapshot: ... |
| 53 | +``` |
| 54 | + |
| 55 | +- Reuses the existing live DB connection (`pg8000`, the driver the integration runner already |
| 56 | + uses) — no new dependency. |
| 57 | +- Because Python has no JDBC `DatabaseMetaData`, it queries the **catalog directly via SQL**, the |
| 58 | + same approach as the TS reference (`introspect/postgres.ts`): `information_schema.columns` / |
| 59 | + `table_constraints` / `key_column_usage` (+ `pg_catalog` where needed for `udt_name`), filtered |
| 60 | + to base tables in `schema`. |
| 61 | +- Returns the existing `SchemaSnapshot` model (`types.py`: `TableDescriptor` → |
| 62 | + `ColumnDescriptor`/`IndexDescriptor`/`FkDescriptor`). **No model changes** — the snapshot shape |
| 63 | + is already shared with `expected_schema` and `diff`. |
| 64 | + |
| 65 | +### 2. Live column type → `SqlType` reconstruction |
| 66 | + |
| 67 | +The reverse of `postgres_emit`'s `SqlType → DDL`. It MUST be the exact inverse so a freshly-emitted |
| 68 | +column introspects back to the same `SqlType` (no phantom diff). Maps the Postgres |
| 69 | +`data_type` / `udt_name` reported by the catalog: |
| 70 | + |
| 71 | +| Postgres type | `SqlType` | |
| 72 | +|---|---| |
| 73 | +| `character varying(n)` / `text` | `Text(n)` / `Text(None)` | |
| 74 | +| `integer` / `bigint` | `Int(32)` / `Int(64)` | |
| 75 | +| `boolean` | `Bool` | |
| 76 | +| `real` (float4) | `Real4` | |
| 77 | +| `double precision` (float8) | `Real` | |
| 78 | +| `numeric(p,s)` | `Numeric(p, s)` | |
| 79 | +| `timestamp without time zone` / `timestamp with time zone` | `Timestamp(with_tz=False/True)` | |
| 80 | +| `date` | `Date` | |
| 81 | +| `uuid` | `Uuid` | |
| 82 | +| `jsonb` | `Json` | |
| 83 | +| `bytea` | `Blob` | |
| 84 | + |
| 85 | +The `uuid` and `jsonb` rows are exactly what Plans 2a/2b need to round-trip; this single mapping |
| 86 | +point serves both the metadata-side (via `expected_schema`) and the live side. It is the Python |
| 87 | +analogue of the **same** fix Plan 2a makes in Java's `JdbcSqlTypes.fromJdbc`. |
| 88 | + |
| 89 | +### 3. Wire into the public API |
| 90 | + |
| 91 | +Export `introspect` from `migrate/__init__.py` alongside `build_expected_schema` / `diff` / |
| 92 | +`emit_postgres`. The `ObjectManager`/migration entry point gains the `expected-vs-live` path: |
| 93 | +`diff(build_expected_schema(root), introspect(conn, schema))` → `emit_postgres(changes)`. This is |
| 94 | +the same three-call shape the other ports expose. |
| 95 | + |
| 96 | +### 4. No behavior change to existing paths |
| 97 | + |
| 98 | +`diff` and `emit_postgres` are unchanged; bootstrap-from-empty still works (empty `actual`). This |
| 99 | +strictly *adds* the missing capability. |
| 100 | + |
| 101 | +## Testing |
| 102 | + |
| 103 | +- **Unit/integration:** a new `test_introspect.py` mirroring Java's `SchemaIntrospectorTest` and |
| 104 | + C#'s `PostgresIntrospectTests` — create tables (covering every `SqlType` above, including a |
| 105 | + `uuid` and a `jsonb` column) against Testcontainers Postgres, introspect, assert the snapshot |
| 106 | + matches; then a **round-trip** test: `emit_postgres(diff(expected, empty))` → apply → |
| 107 | + `introspect` → assert `introspect(...) == expected` (no phantom diff). |
| 108 | +- **Parity check:** an `expected-vs-live` convergence test — bootstrap, then diff a modified |
| 109 | + expected schema against the introspected live one, asserting only the delta is emitted (the |
| 110 | + capability the other four ports already have). |
| 111 | +- TDD throughout; pre-merge gate = code review + simplify before merge-forward. |
| 112 | + |
| 113 | +## Per-port touch summary |
| 114 | + |
| 115 | +Python only — the other four ports already have introspection. This closes the parity gap so all |
| 116 | +five share the same introspect → diff → converge capability. |
| 117 | + |
| 118 | +## Consequences |
| 119 | + |
| 120 | +- Removes the "Python out of scope" caveat from Plans 2a and 2b — uuid/jsonb/timestamptz columns |
| 121 | + round-trip via introspection in Python too. |
| 122 | +- Python's migrate engine reaches capability parity with TS/C#/Java (Kotlin uses Exposed's own |
| 123 | + schema management and is parity-exempt by design). |
| 124 | +- The live type → `SqlType` mapping is the inverse of `postgres_emit`; the two must stay in sync |
| 125 | + (a round-trip test enforces it). |
| 126 | + |
| 127 | +## Out of scope (explicit) |
| 128 | + |
| 129 | +- SQLite introspection for Python (the corpus is Postgres-only; SQLite is a TS-only dialect). |
| 130 | +- Index/FK introspection fidelity beyond what `diff` consumes — match the existing |
| 131 | + `ColumnDescriptor`/`IndexDescriptor`/`FkDescriptor` fields the other ports populate; do not |
| 132 | + expand the model. |
0 commit comments