fix(store): ADR 0114 sub-lever A — the claim-proc startup gate has never passed

[wshallwshall]

The bug

OBJECT_DEFINITION() does not return a CREATE OR ALTER module verbatim. SQL Server deletes the OR and ALTER keyword tokens and keeps their separators, so a head submitted as CREATE OR ALTER PROCEDURE dbo.x comes back as CREATE + three spaces + PROCEDURE dbo.x — character delta exactly 7, everything after the head byte-identical.

The ADR 0114 startup gate hashed the submitted text. It therefore could never match, and sub-lever A has been inert in every deployment since it shipped — fifo_claim_proc degraded to the ad-hoc batch on every open, silently, for the whole life of the feature.

Measured on SQL Server 2022 16.0.4255.1 and 2025 17.0.4055.5, compat 130/160/170, across five deploy paths (fresh CREATE, the OR ALTER re-apply, a plain batch, inside the shipped guarded EXEC(N'…'), and an out-of-band ALTER PROCEDURE — which the engine also rewrites, to a single-spaced CREATE PROCEDURE).

Live legs: 5 failed / 1 passed → 6 passed.

Why the tests didn't catch it

_gate_rows resolved its "SHIPPED" sentinel by calling ss._claim_proc_body(...) and handing that back as the fake OBJECT_DEFINITION answer — modelling the server as an identity function. Both sides of the gate's comparison were the same function of the same argument. Three mutually contradictory implementations (the bug, a correct fix, and an over-wide fix that blinds the gate to the whole signature) all scored 51/51 against it.

There was a second door: test_ac7_gate_tolerates_whitespace_differences bypassed the fixture default by passing cid_body= explicitly, with a string that still carried CREATE OR ALTER. Fixing only the default would have left the suite blind in exactly the same way.

Both are closed here.

The fix — shipped-side only

The gate now compares the deployed hash against a small set of code-controlled constants: the head forms a server may store for a module this build deployed (_CLAIM_PROC_STORED_HEADS — the measured rewritten form, plus verbatim for a hypothetical non-rewriting engine, which is simply the text we submitted).

_claim_proc_body() renders byte-identically, so _claim_proc_ddl, _SCHEMA, _schema_hash() and both golden body pins are untouched — no re-pin, and no forced DDL re-apply on any live database. Verified: the body hashes are still 9c685181… / 0eaf2de2….

Three design points worth review:

• Keyed per proc, deliberately. A single flat accepted-set would take the cid body served under the dst name — reachable with no tampering intent via sp_rename, which does not rewrite sys.sql_modules.definition — and silently swap the lane predicate, claiming zero rows forever.
• Head spellings this deploy path cannot emit keep failing the gate (CREATE PROC, differing case). Each is affirmative evidence of an out-of-band hand deploy, which is the AC-7 event. A two-sided canonicalization would launder exactly that signal.
• Not a regex. A two-sided regex form was evaluated and rejected: its correctness rests on properties no finite assertion can pin. Mutation-scored 0/7 against this suite — dropping ^, adding MULTILINE, dropping count=1, widening to \w+, eating the object name, dropping IGNORECASE all survive every test, and one of them makes the gate accept a body deployed under the other lane family's name. The shipped-side form is pinned by one set-equality assertion.

Two more fixes, in the same PR because the gate fix makes them reachable

The pin leak — this is why D1 must not land alone

The proc CALL pins 9 parameter descriptors on the pooled cursor (descriptor[0] = SQL_DOUBLE for @now FLOAT). setinputsizes is persistent cursor state, and at OUTBOUND the same cursor then runs the H2 delivery probe, binding an NVARCHAR id against the stale SQL_DOUBLE descriptor → client 22018 → rollback → outbound delivery collapse (measured previously at 1122 errors == 1122 failed deliveries). Pins are now cleared the moment the CALL's result is drained.

main today ships the lever but not the fixes. A prior cherry-pick landed the ADR 0114 flag architecture while reverting this and the gate fix. Flags default OFF so it is inert — but anyone setting fifo_claim_proc=true on current main gets the dead gate and the collapse. Fixing the gate alone would arm the collapse for the first operator who flips the flag.

CI could never have caught it. Every test in test_adr0114_claim_proc_live.py claims at Stage.INGRESS; the only OUTBOUND leg that exists anywhere is the one this PR adds. So landing the gate fix and wiring the file — the obvious plan — would have turned that file green in CI while the collapse stayed live in production, with nothing downstream able to see it.

Measured on pyodbc 5.3.0 / ODBC Driver 18: a zero-parameter execute tolerates surplus descriptors, so the shielded SET LOCK_TIMEOUT -1; reset in the finally guard is unaffected and the exposure is exactly the H2 bind chain. That measurement carries its own positive control — the same run reproduced [22018] Invalid character value for cast specification (0) on the parameterized bind, where (0) is the SQL_DOUBLE @now descriptor. A recorded negative with a demonstrated positive control, so nobody has to re-open the question.

AC-11 no-match parity on the ad-hoc batch

Only the two flagged branches honoured it. The batch bound the raw lane list into (VALUES (?),…) feeding a DECLARE @heads TABLE (lane NVARCHAR(256) NOT NULL — SQL Server evaluates that narrowing conversion on the outer constant scan before the CROSS APPLY filters it, so with ANSI_WARNINGS ON an oversized lane raises 2628 even when zero rows would match. 2628 is not 1222, so it is not translated to EMPTY-all: it rolls back and re-raises to the dispatcher.

This was not deferrable — fixing the gate made the parity test reach its batch arm and it failed live. The skip is hoisted ahead of the dispatch-path split so all three branches agree. The clamp still runs before the skip, preserving the tested contract (test_prepared_lane_encoding_shares_the_proc_rules pins 499, not 500).

Verification

check	result
Full offline suite	9490 passed, 825 skipped, 0 failed
Live legs, real SQL Server 2022 CU25	6 passed (was 5 failed / 1 passed)
Mutation — bug / flattened key / over-wide / pin-clear no-op	10 / 3 / 8 / 1 failed
Mutation — correct implementation	64 passed
ruff · ruff-format · mypy · ledger gate · bandit · secrets	pass
Golden body pins	unmoved

The mutation row is the point: the previous suite could not distinguish a working gate from a broken one. Every wrong implementation is now caught by a named test.

Docs

Three docstrings asserted the false premise ("OBJECT_DEFINITION() preserves the definition text as executed"). The operator-facing degraded reason prescribed DELETE FROM schema_meta, which could not work — the re-apply submits the same text, the engine rewrites it identically, and the hash mismatches again, forever. Both corrected, plus an ADR 0114 amendment.

AC-7b added. AC-7 is one-directional: it required the gate to degrade on a mismatch, and nothing anywhere required a correctly deployed proc to pass. The defect therefore satisfied AC-7 literally while leaving the lever inert — AC review could not have caught this. Any future gate-shaped AC needs both directions.

CI

Adopts the dormant ci-suite-wiring commit (cherry-picked with -x), which wires 16 module-gated suites that ran in no workflow step — 84 tests, dark — into four new steps, and adds adr0114 to the path-filter alternation so a test-only follow-up still fires the leg. Includes test_serverdb_ci_coverage.py, which fails if a module-gated suite is added without wiring.

Performance

Latency, not capacity. Previously measured at ~65% claim_mean reduction (~9 ms/claim). Sustained throughput is unchanged — the characterisation of record is "latency-proven, throughput-null on N=4", sourced to the at-knee soak (336.3 vs 337.1, arm-identical) and TO-ENGINE-072, not to the @150/180 re-gate, whose rungs were sub-knee and explicitly declined to project. No throughput claim is made and no published capacity figure changes.

Not in this PR — recommended follow-ups, not yet filed

Deliberately out of scope here. Both need a BACKLOG number, and I have not allocated one: the allocator mutates a shared ledger, several sessions are live, and number reuse has corrupted that ledger before. Flagging them for the owner to file (or to tell me to) rather than allocating speculatively mid-review.

1. The AC-7 degraded gauge does not exist. claim_proc_effective and claim_proc_degraded_reason reach no API, /stats, /metrics or console surface — grep returns only store/sqlserver.py and tests. ADR 0114:401 describes the gate as setting a degraded gauge; it does not. The sole operator signal is one WARNING line at open. That is a real part of why this defect ran unnoticed for the life of the feature, and it is not fixed here.

2. A missing VIEW DEFINITION grant is misdiagnosed as a missing proc. A principal holding only EXECUTE sees OBJECT_ID non-NULL but OBJECT_DEFINITION NULL. The gate treats a NULL body as missing and reports "guarded DDL skipped — CREATE PROCEDURE / ALTER-on-schema denied", sending the operator to entirely the wrong control; the cure is GRANT VIEW DEFINITION. WITH ENCRYPTION produces the same NULL and the same misdiagnosis. This is not hypothetical for this codebase — the sub-lever B comment at sqlserver.py:904-905 explicitly designs for a fleet whose principal can never hold CREATE PROCEDURE, which is exactly the posture that hits it. Fix: distinguish OBJECT_ID IS NULL from OBJECT_ID NOT NULL AND OBJECT_DEFINITION IS NULL and name the right grant. I left the probe SQL unchanged here so the gate's stubbed-_fetchone contract stays pinned.

Also deferred: the AC-11 lane-skip ordering question (the clamp currently runs before the skip, so an unmatchable lane still consumes a chunk slot — changing it would serve more lanes per call but alters a tested contract), and the orphaned vaultall/vault ref namespace, which holds ADR 0116, ADR 0117, two features and 22 BACKLOG items invisible to any normal git query. A separate session is surveying that.

🤖 Generated with Claude Code