fix(2pc): clean up prepared transactions recorded in the WAL after a restart

[willothy]

Problem

With the 2pc WAL enabled, transactions in flight during a SIGTERM/SIGKILL were left as orphaned prepared transactions on the shards after PgDog rebooted, even though WAL recovery ran.

Since #1248, prepared transaction GIDs embed instance_id(), which (without NODE_ID) is a random string generated per process. The WAL only persisted the numeric transaction ID, so after a restart, cleanup re-rendered the GID with the new process's instance ID and issued ROLLBACK PREPARED / COMMIT PREPARED against a name that doesn't exist. The 42704 ("prepared transaction does not exist") skip in two_pc_on_guards made that look like success, so cleanup wrote End records to the WAL and permanently hid the orphans. Undecided transactions kept holding locks. Decided ones were left half-committed.

Solution

Record the coordinator GID prefix in every WAL Begin record (and checkpoint entry), so recovery knows the exact prepared transaction names on the shards. The field is additive (#[serde(default)]); no tag or segment format changes.

Make Binding::two_pc_cleanup scan pg_prepared_xacts on each shard (filtered to current_database()) and resolve the transaction by its exact recorded GID. The scan is what makes retries idempotent: a resolved GID stops appearing, so nothing is re-committed or re-rolled-back. A GID that shares the numeric ID under a different prefix belongs to another coordinator and is never touched. For WAL records written by versions that did not store the prefix, matching falls back to the durable numeric transaction ID plus shard index; a wrong-target action through the fallback requires a 64-bit random ID collision between concurrently active transactions. Manager::cleanup_phase uses this for all cleanup (WAL recovery, interrupted clients, replication copy).

Supporting hardening:

• TwoPcTransaction::from_str now requires the __pgdog_2pc_ prefix, so cleanup can never touch another application's prepared transactions.
• GIDs are acted on only if they contain nothing but [A-Za-z0-9_-]; anything else is refused with a warning. NODE_ID and DEPLOYMENT_ID are validated against the same alphabet at startup (empty now behaves as unset, like HOSTNAME). This makes the GIDs safe to embed in transaction control statements without escaping.
• COMMIT/ROLLBACK PREPARED failing with 42501 (insufficient privilege) warns with the exact statement for the operator instead of retrying forever and replaying on every restart.

Testing

• Live regression tests: exact-GID resolution under a previous process's recorded prefix (test_cleanup_transaction_foreign_prefix, fails on main); numeric fallback for legacy records without the prefix field; and a structural-guarantee test proving cleanup leaves a same-number, different-prefix GID untouched.
• WAL format tests: round-trips with the new field, and the old record shape (no prefix key) decodes with an empty prefix.
• integration/two_pc_crash_safety now passes all 4 scenarios, including kill-mid-PREPARE followed by a restart with a fresh instance ID (this is the reported scenario, and it fails on main). Its synthesized-WAL commit test writes records without the prefix field, covering the legacy fallback end to end. Its wal_helper no longer compiled (default-features = false vs pgdog-plugin's pg_query requirement); fixed here. Note this suite is not wired into CI or integration/run.sh, which is how the regression shipped unnoticed; adding it to CI would be a good follow-up.
• Unit coverage for GID matching (foreign/deployment prefixes, wrong shard/ID, non-PgDog GIDs, unsafe alphabets) and identifier validation. Full suite passes.

Known limitations

• WALs written by affected versions (v0.1.48+) already contain End records from the silently-failed cleanups, so existing orphans won't self-heal; they need one manual sweep of __pgdog_2pc_* entries in pg_prepared_xacts.
• Existing deployments using invalid node IDs will error.

Closes #1175

[codecov]

Codecov Report

❌ Patch coverage is 96.65810% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pgdog/src/backend/pool/connection/binding.rs	88.52%	7 Missing ⚠️
.../frontend/client/query_engine/two_pc/wal/writer.rs	0.00%	3 Missing ⚠️
...rontend/client/query_engine/two_pc/wal/recovery.rs	84.61%	2 Missing ⚠️
...src/frontend/client/query_engine/two_pc/manager.rs	93.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!