fix: make transform.sh patches fail loudly

[NikolayS]

Build-pipeline integrity fixes (transform.sh)

Addresses findings C1, C2, C3, C5, C8 of #283.

Bugs and fixes

C1 — silent no-op sed patches (medium)

The critical correctness patches (batch_event_sql xid8 quoting, batch_retry NULL::int8 → NULL::xid8, upgrade_schema return 0 → return cnt, sqltriga header fix, annotation seds) were plain sedi calls: sed exits 0 whether or not it matched, and the script printed PASS: ... unconditionally. If upstream PgQ reworded an anchored line, the build would report success while the generated batch_event_sql emits unquoted txids → runtime operator does not exist: xid8 = integer on every batch fetch.

Fix: new sedi_must helper (cmp before/after; exits 1 with the failing expression and file when nothing changed). Every must-match sedi now goes through it. The upgrade_schema role-grant and insert_event_raw guard awk patches now count their substitutions/injections and fail on a count mismatch, matching the standard already set by the get_batch_cursor patch.

C2 — missing source files only warned (medium)

• Transform loop: WARNING ... continue → hard error (exit 1).
• FUNCTION_FILES assembly loop: WARNING → hard error.
• pgque-api files: previously silently skipped with no message → hard error; a missing sql/pgque-api/ directory is also a hard error.
• pgque-additions files: explicit hard error with a clear message.

A submodule bump that renames/drops a file can no longer ship a truncated install file that passes the CI drift check after regenerate-and-commit.

C3 — vacuous pg_notify check + unverified ticker injection (medium)

• (a) The ticker pg_notify awk injection now counts both anchors (return i_tick_id;, return currval(q.queue_tick_seq);) and fails the build unless each fired exactly once. Same for the create_queue length-check injection.
• (b) The post-assembly check grep -q 'pg_notify' "$INSTALL_FILE" was vacuous — Section 6 (hardening.sql) contains its own pg_notify, so it passed even if BOTH ticker injections missed. Replaced with an awk that extracts each tick-emitting pgque.ticker definition from the generated file and requires perform pg_notify( inside each body.

C5 — bogus annotation on hardening.sql's pg_notify line (low, changes generated files)

The -- PgQue transformation: LISTEN/NOTIFY wakeup (not in original PgQ) annotation sed ran against the whole assembled file and also tagged the pg_notify line inside hardening.sql's external-ticker override — a line that was never in PgQ — making the embedded chunk differ from sql/pgque-additions/hardening.sql.

Fix: the annotation block now runs before Sections 6/7 are appended, so only the transformed-PgQ portion is annotated. Regenerated files: the only change to sql/pgque.sql and sql/pgque-tle.sql is that line losing the bogus trailing comment, making the embedded hardening chunk byte-identical to its source (verified with a chunk diff, see below).

C8 — hardcoded header version (low)

The generated header hardcoded -- Version: 0.2.0 while the pg_tle registration version was extracted dynamically from pgque.version() in lifecycle.sql. A version bump in lifecycle.sql would leave the header stale with no check.

Fix: the version is extracted once, early (same anchored awk + semver validation), into PGQUE_VERSION and reused for both the header and the pg_tle registration. Current version is 0.2.0, so regeneration does not change the version text.

Verification

1. Build on this branch: exit 0, expected diff only

$ bash build/transform.sh; echo exit=$?
...
=== ASSEMBLY COMPLETE — ALL CHECKS PASSED ===
=== pg_tle PACKAGING COMPLETE ===
exit=0

$ git diff --stat   # (before commit)
 build/transform.sh | 361 ++++++++++++++++++++++++++++++++++++++---------------
 sql/pgque-tle.sql  |   2 +-
 sql/pgque.sql      |   2 +-

The full generated-file diff (the single C5 annotation line, once per file):

--- a/sql/pgque.sql
+++ b/sql/pgque.sql
@@ -5187,7 +5187,7 @@ begin
-    perform pg_notify('pgque_' || i_queue_name, i_tick_id::text); -- PgQue transformation: LISTEN/NOTIFY wakeup (not in original PgQ)
+    perform pg_notify('pgque_' || i_queue_name, i_tick_id::text);
--- a/sql/pgque-tle.sql
+++ b/sql/pgque-tle.sql
@@ -5275,7 +5275,7 @@ begin
-    perform pg_notify('pgque_' || i_queue_name, i_tick_id::text); -- PgQue transformation: LISTEN/NOTIFY wakeup (not in original PgQ)
+    perform pg_notify('pgque_' || i_queue_name, i_tick_id::text);

2. Negative tests (submodule restored with git -C pgq checkout . after each; nothing committed)

C1 — reword the batch_event_sql anchor line:

$ sed -i "s/|| ' and ev.ev_txid >= ' || batch.tx_start::text/|| ' and ev.ev_txid>=' || batch.tx_start::text/" \
    pgq/functions/pgq.batch_event_sql.sql
$ bash build/transform.sh; echo exit=$?
...
FAIL: batch_event_sql xid8 quoting (tx_start lower bound) did not apply
      sed expression matched nothing in .../build/output/functions/pgque.batch_event_sql.sql:
      s/|| ' and ev.ev_txid >= ' || batch.tx_start::text/|| ' and ev.ev_txid >= ''' || batch.tx_start::text || '''::xid8'/
      Upstream PgQ likely reworded the anchor line; update
      build/transform.sh to match the new source.
exit=1

C2 — delete an expected source file:

$ rm pgq/functions/pgq.batch_retry.sql
$ bash build/transform.sh; echo exit=$?
ERROR: source file not found: functions/pgq.batch_retry.sql
       A submodule bump may have renamed or dropped it; update
       SOURCE_FILES in build/transform.sh before rebuilding.
exit=1

C2 — remove a pgque-api file (previously silently skipped):

$ mv sql/pgque-api/send.sql /tmp/ && bash build/transform.sh; echo exit=$?
ERROR: pgque-api file not found: pgque-api/send.sql
       Shipping without it would truncate the install file.
exit=1

C3 — break the ticker-injection anchor indentation:

$ sed -i 's/^    return i_tick_id;$/        return i_tick_id;/' pgq/functions/pgq.ticker.sql
$ bash build/transform.sh; echo exit=$?
ERROR: ticker pg_notify injection anchors matched 0/1 times (expected 1/1)
FAIL: pg_notify injection into ticker did not happen; anchor lines
      in pgq/functions/pgq.ticker.sql may have changed upstream.
exit=1

C3b — the new post-assembly check is non-vacuous. Against a doctored copy of sql/pgque.sql with the two injected ticker pg_notify lines stripped (hardening.sql's own pg_notify still present — this fooled the old whole-file grep, grep -c 'perform pg_notify' = 1):

ERROR: pg_notify missing from ticker bodies (4-arg: 0/1, 1-arg: 0/1)
check exit=1

C8 — version propagation (temporary bump of lifecycle.sql to 9.9.9, then restored):

$ grep -n -- '-- Version:' sql/pgque.sql | head -1
2:-- Version: 9.9.9
$ grep -n '9.9.9' sql/pgque-tle.sql | head -1
71:    if existing_version = '9.9.9' then

3. Fresh install + full test suite

$ createdb pgque_build_oqxpbr
$ psql -X -v ON_ERROR_STOP=1 -d pgque_build_oqxpbr -f sql/pgque.sql      # exit 0
$ psql -X -v ON_ERROR_STOP=1 -d pgque_build_oqxpbr -f tests/run_all.sql  # exit 0
...
=== ALL TESTS PASSED ===

4. Hardening chunk byte-identity

$ diff <(extracted pgque-additions/hardening.sql chunk from sql/pgque.sql) sql/pgque-additions/hardening.sql
CHUNK BYTE-IDENTICAL

Manual verification command

bash build/transform.sh && git diff --exit-code sql/ \
  && createdb pgque_scratch && psql -X -v ON_ERROR_STOP=1 -d pgque_scratch -f sql/pgque.sql \
  && psql -X -v ON_ERROR_STOP=1 -d pgque_scratch -f tests/run_all.sql

Note: other branches from #283 also regenerate sql/pgque.sql / sql/pgque-tle.sql; merge conflicts there are expected and should be resolved by re-running build/transform.sh. The TLE header wording about client compatibility (finding C7) is intentionally untouched here.

https://claude.ai/code/session_01KAaEGkQZmey1D1xCsVGmqv

---

Generated by Claude Code