feat(pg): PostgreSQL datastore compatibility layer

[dnplkndll]

Related issue: internal PG-compat fork work (no upstream issue)

Checklist for submitter

• Input data is properly validated, SELECT * is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.
• Timeouts are implemented and retries are limited to avoid infinite loops

Testing

• Added/updated automated tests
• Where appropriate, automated tests simulate multiple hosts and test for host isolation (updates to one hosts's records do not affect another)
• QA'd all new/changed functionality manually

Database migrations

• Checked schema for all modified table for columns that will auto-update timestamps during migration.
• Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects.
• Ensured the correct collation is explicitly set for character columns (COLLATE utf8mb4_unicode_ci). (N/A on PG; MySQL paths unchanged.)

---

What changed

Full PostgreSQL compatibility for Fleet's datastore layer, enabling the production deployment at fleet.hz.ledoweb.com to run against PostgreSQL 16 instead of MySQL.

Current state (2026-07-28): the live branch is feat/pg-compat-rebased, rebased onto upstream fleetdm/fleet@5cd8edbfb5. A full adversarial review (2026-07-27) was remediated across three phases — honest test gate, schema parity repair, driver/runner hardening — and a follow-up adversarial re-review found 3 more must-fixes, all closed the same day: the swap-caller bare DROP TABLE (broke the vulnerability host-counts cron hourly), the Windows ESP tri-state collapse, and updated_at frozen on ~125 tables (now a generated 138-trigger set applied on every prepare db with exact MySQL semantics). See docs/Deploy/pg-review-remediation.md for the complete findings reconciliation. Five validators + three staleness-checked generators gate every push. A branch quality pass (DRY/coverage/docs) followed. Phase 4 (the declared merge gate) is COMPLETE (2026-07-29): the PG CI job runs the datastore package with no -run filter — 121 dual-dialect tests pass against PG, 59 MySQL-only tests skip visibly, zero failures. Mid-phase the branch was re-rebased onto upstream fleetdm/fleet@54b32ddf88 (~84 more commits, conflict-free); two upstream migrations timestamped below our baseline marker were caught by the below-marker drift backstop and ported via Migration 20260729190000 (with a post-condition assert; one insert inlined because MySQL's integer literal for a boolean column is a 42804 on PG). Baseline marker: 20260729190000.

July 2026 rebase onto upstream 5cd8edb (1,114 upstream commits)

• Re-resolved the PG layer across upstream's June–July refactors (Windows MDM reconciler rewrite, policy exclude-all scoping, FMA canonical naming — which upstreamed this fork's inline rename feature, software-title singleflight inserts, DDM scoped declarations).
• 14 new upstream migrations ported to PG with isPostgres() branches: ALTER … CHANGE/MODIFY, DROP CHECK, inline ADD [UNIQUE] KEY, combined DROP/ADD INDEX, VIRTUAL generated ENUM columns (plain text on PG), IF() in CHECK expressions, REGEXP → ~, MySQL information_schema FK/index lookups → pg_constraint/pg_indexes. Idempotency guards added for schema this fork shipped ahead of upstream (pending-delete table, policy_gated, BYOD/ADUE).
• PG tests now replay post-baseline migrations exactly as fleet prepare db does (seed history ≤ marker, goose Up through the rebind driver), so migration incompatibilities surface in TestPostgres* instead of at deploy.
• Baseline regenerated from a scratch PG migrated via fleet prepare db --mysql_driver=postgres; marker bumped to 20260724134801. schema_identity_cols_gen.go regenerated; schema/column drift validators clean.
• Rebind driver: strips ON UPDATE NOW(n) like ON UPDATE CURRENT_TIMESTAMP (emitting the updated_at trigger); ten new knownPrimaryKeys entries for new upstream upsert sites; the two mdm_configuration_profile_variables upserts (per-kind conflict targets) converted to the dialect helper.
• GetHostMDM fix: upstream's new connected_to_fleet CASE mixed EXISTS(...) with 1/0 integer literals — PG rejects mixed CASE branch types (SQLSTATE 42804), which broke /api/fleet/orbit/config on first deploy. Converted to true/false literals (valid in both dialects); regression-covered by TestPostgresGetHostMDM.
• Vulnerabilities cron fix: InsertCVEMeta on PG adds a WHERE … IS DISTINCT FROM guard so re-upserting the ~300k-row NVD set doesn't rewrite unchanged rows (dead tuples + index churn), and LoadCVEMeta's bulk-load deadline is raised 1m → 10m. This unblocks the hourly insert cve scores: context deadline exceeded failure observed in prod since at least Jul 12.

Core dialect abstraction

• DialectHelper interface (dialect.go) abstracting all MySQL vs PostgreSQL SQL differences
• mysqlDialect and postgresDialect implementations covering all methods: InsertIgnoreInto, ReplaceInto, FromDual, OnDuplicateKey, OnConflictDoNothing, GroupConcat, JsonQuote, JSONAgg, JSONExtract, JSONUnquoteExtract, JSONBuildObject, JSONObjectFunc, FindInSet, FullTextMatch, RegexpMatch, GoquDialect, ReturningID, IsPostgres, CreateTableLike, AtomicTableSwap

Runtime SQL translation (server/platform/postgres/rebind_driver.go)

The pgx-rebind driver wraps pgx/stdlib and rewrites MySQL syntax to PG at the driver layer:

• Placeholders: ? → $N
• Boolean columns: col = 1/col = 0 → col = true/col = false (for the ~60 boolean cols listed in schema_bool_cols_gen.go)
• JOIN syntax: UPDATE … JOIN → UPDATE … FROM; multi-table DELETE → DELETE … USING
• JSON functions: JSON_EXTRACT(col, '$.path') → col->'path'; JSON_OBJECT(…) → jsonb_build_object(…); JSON_QUOTE(…), JSON_UNQUOTE(…), JSON_ARRAYAGG(…) → PG equivalents
• MySQL functions: IF() → CASE WHEN, IFNULL → COALESCE, MD5() → md5(), UUID() → gen_random_uuid()::text, UTC_TIMESTAMP() → TO_CHAR(NOW() AT TIME ZONE 'UTC', …), CURDATE() → CURRENT_DATE, DATABASE() → current_schema(), HEX()/UNHEX() → encode/decode, FIND_IN_SET → array_position(string_to_array), …
• Type casts: CAST AS UNSIGNED → CAST AS integer, CAST AS SIGNED INT → CAST AS integer, TIMESTAMP(?) → (?)::timestamp
• Upsert: INSERT IGNORE INTO → INSERT INTO … ON CONFLICT DO NOTHING; REPLACE INTO → INSERT … ON CONFLICT DO UPDATE; ON DUPLICATE KEY UPDATE VALUES(col) → ON CONFLICT (pk) DO UPDATE SET col = EXCLUDED.col
• DDL column-type translation: BLOB → bytea, MEDIUMTEXT/LONGTEXT → TEXT, TINYINT(1) → smallint, DATETIME[(N)] → timestamp[(N)], INT UNSIGNED NOT NULL AUTO_INCREMENT → INTEGER NOT NULL GENERATED BY DEFAULT AS IDENTITY, INT UNSIGNED → INTEGER, inline UNIQUE KEY name (cols) → CONSTRAINT name UNIQUE (cols), enum('a','b','c') → VARCHAR(255) CHECK (col IN ('a','b','c')), ENGINE=InnoDB/DEFAULT CHARSET=…/ALGORITHM=INSTANT stripped
• DDL multi-statement output: ALTER TABLE … ADD COLUMN …, ADD KEY <name> (<cols>) produces an ALTER followed by a separate CREATE INDEX. ADD UNIQUE KEY → CREATE UNIQUE INDEX. Multiple ADD KEY clauses each become their own CREATE INDEX.
• ON UPDATE CURRENT_TIMESTAMP / NOW(n): the MySQL column attribute is stripped from CREATE TABLE and a per-table BEFORE UPDATE trigger calling fleet_set_updated_at() is appended. The trigger function is installed by pg_baseline_post.sql.

Migration runner

• MigrateTables on PG: applies the embedded baseline on fresh DBs, seeds migration_status_tables/migration_status_data ≤ the baseline marker, then runs goose Up so newer upstream migrations execute through the rebind driver.
• pg_baseline_post.sql: idempotent ownership fixups + fleet_set_updated_at() trigger function; re-applied on every prepare db.
• Migrations that MySQL expresses as generated-column redefinitions are modeled on PG as plain columns with one-time stored-value recomputes.

pgcompat tooling (tools/pgcompat/)

Validators that gate every push:

• check_primary_keys — every raw ON DUPLICATE KEY UPDATE site is covered by knownPrimaryKeys, and every entry's columns match a real PK/UNIQUE constraint in the baseline (also run with --include-migrations)
• check_schema_drift / check_column_drift — schema.sql vs pg_baseline_schema.sql table and column sets match (allowlists for intentional drift)
• check_constraint_drift — PK/UNIQUE/index/FK parity by (table, kind, column-set); the deferred ~160-FK set is allowlisted with rationale
• check_bool_col_split — no column name typed boolean and smallint in different tables (split names are excluded from all name-keyed rewrites)
• gen_updated_at_triggers — generates the ON UPDATE CURRENT_TIMESTAMP trigger mirror (138 triggers) applied on every prepare db; CI-staleness-checked like gen_bool_cols / gen_identity_cols
• gen_bool_cols / gen_identity_cols — regenerate the schema-derived driver tables; CI fails if stale
• Fresh-PG-install smoke test: empty PG + prepare db (baseline + post-marker migrations), asserted idempotent on second run

Test plan

• Validate PG Compatibility CI — green on feat/pg-compat-rebased @ dd8ae2b and on aggregated
• Go Tests (PostgreSQL) CI — green on aggregated
• Build & Push Ledo Fleet Image CI — green; image sha256:6680b5b9eefd… in registry.hz.ledoweb.com
• Local: full TestPostgres* suite against real PG 16, including post-baseline migration replay and the new TestPostgresInsertCVEMeta
• Local: MySQL TestMigrations passes (MySQL paths of ported migrations unchanged)
• fleet prepare db --mysql_driver=postgres migrates a scratch PG cleanly through all July migrations (used to regenerate the baseline)
• Prod DB migrated (all 27 post-baseline migrations applied cleanly via fleet prepare db job, 2026-07-27) and rebased image rolled out
• Post-deploy verification: orbit config errors stopped, hourly cve_meta timeout gone, CVE/EPSS data refreshing (verified 2026-07-27)
• Review-remediation phases 1–3 deployed to prod with per-phase verification (migrations, backfills, index parity, swap-name canonicalization, device-endpoint walks, zero error logs)
• Phase 4: Go Tests (PostgreSQL) CI runs the datastore package unfiltered — 121 pass / 59 MySQL-only skips / 0 fail; driver + goose + pgcompat suites green; fresh + idempotent prepare db verified post-rebase onto 54b32ddf88
• Post-deploy adversarial re-review of the Phase-4 delta (2026-07-29, three parallel reviewers): all findings fixed in-branch — 084359's integer boolean literal (a deploy wedge for PG DBs still below that version), the dedup migration's missed policies.patch_software_title_id and unguarded software_title_team_pins repoints, intra-loser slot collisions in the guarded repoint pattern, a pre-flight guard for the bundle-identifier unique index the backfill newly activates, both-table guards + per-row idempotent variable inserts in the porting wrapper, TranslateError unit coverage, and trigger-formula parity tests for all four generated columns. Prod audited for the missed-table gap: zero orphans (no patch policies exist there)
• Phase 4 deployed to prod (2026-07-29): the two failed migration attempts each hardened the branch — the below-marker backstop now admits declared porting wrappers (PortedBelowMarker), and the generated-column backfill dedups the 95 duplicate software_titles rows that stale unique_identifier values had let accumulate (rehearsed against prod data in a rolled-back transaction first). Post-deploy: max version 20260729190000, 0 dup groups, 0 orphans, 138 updated_at-trigger tables, crons completing, hosts checking in