ADR 0114 follow-ups: the AC-7 degraded gauge, and VIEW DEFINITION misdiagnosed as a missing proc

docs/adr/0114-phase-4-claim-path-call-complexity-reduction-driver-interface-redesign-ingress-routed-reset-fold.md

@@ -398,9 +398,9 @@ compat-120 database and under a DDL-denied principal.
 `OBJECT_ID` of **both** procs; (b) a SHA-256 of each deployed body via `OBJECT_DEFINITION()` against the
 **stored forms** of the shipped DDL text (normalized) — **existence alone cannot catch a hand-edited body**, and
 the ADR 0064 marker covers only in-repo edits, while the proc *is* the claim logic; (c) `compatibility_level ≥
-130`. Any failure → the store records `claim_proc_effective = False`, logs a **WARNING naming the reason** and
-runs the shipped batch — never a lane outage; the hot path contains **no error-2812 handling**. Out-of-band
-drift is caught at the next open.
+130`. Any failure → the store records `claim_proc_effective = False`, logs a **WARNING naming the reason**,
+publishes the degraded gauge (see the second amendment below), and runs the shipped batch — never a lane
+outage; the hot path contains **no error-2812 handling**. Out-of-band drift is caught at the next open.
 
 > **AMENDMENT (2026-07-30) — `OBJECT_DEFINITION()` does not return the submitted text, and this gate was
 > inert until it was fixed.**
@@ -437,6 +437,65 @@ drift is caught at the next open.
 > work** — the re-apply submits the same text, the engine rewrites it the same way, and the hash mismatches
 > again — so the advice has been removed from the ADR and from the operator-facing degraded reason.
 
+> **AMENDMENT (2026-07-31) — the degraded gauge now exists, and probe (a) is a real probe again.** Two
+> follow-ups the amendment above deliberately held out of the bug fix.
+>
+> **1. The gauge was aspirational.** AC-7 requires "a WARNING naming the reason **+ degraded gauge**", and this
+> section's compensating-control story assumes an operator can SEE the degraded state. Until this amendment
+> nobody could: `claim_proc_effective` / `claim_proc_degraded_reason` were read by the store's own tests and
+> **nothing else** — no `/stats`, no `/status`, no `/metrics`, no console. The entire operator signal was one
+> WARNING line at `open()`. That is not a missing nicety, it is a load-bearing part of *why the amendment above
+> was needed*: a fleet running the flag degraded on every open, forever, and the only thing that could have
+> told anyone was a log line nobody was watching which named the wrong cause.
+>
+> The gauge is now a store accessor, `claim_proc_status()`, surfaced on three operator surfaces:
+>
+> | surface | carries |
+> |---|---|
+> | `GET /status` → `claim_proc` | `effective`, the human-readable `degraded_reason`, and the matched `head_forms` |
+> | `GET /metrics` | `messagefoundry_store_claim_proc_effective` (0/1) and `messagefoundry_store_claim_proc_head_verbatim` (0/1) |
+> | the console's store panel (`/ui/status`) | active-vs-degraded, plus the reason when degraded / the head forms when green |
+>
+> Three shape decisions, so they are not re-litigated. **`None` when the flag is off**, so "not requested" is a
+> distinct state from "requested and degraded"; the Prometheus series are correspondingly **absent**, not a
+> constant `0` that every SQLite fleet would publish unalertably. **No reason label in the exposition** — the
+> reason is free text embedding a proc name and, on the probe-failure arm, an exception string, so a label
+> would be unbounded cardinality *and* a breach of the exporter's strict `{connection, destination, status,
+> version, le}` allowlist; the string lives on `/status` and the console instead. **It does not feed the
+> console's engine-health heart**: a degrade is a performance lever not paying off, claims keep flowing, and
+> making the nav cry wolf about it would devalue the signal that means the store is actually unwell.
+>
+> `head_forms` (proc name → `rewritten` | `verbatim`) is surfaced for the same reason it is logged: a fleet
+> reporting `verbatim` is a live counterexample to `_CLAIM_PROC_STORED_HEADS`'s compatibility assumption — no
+> engine measured to date stores the `CREATE OR ALTER` head unrewritten — and it was previously visible only
+> at INFO. Observability only: the accept/degrade logic is untouched.
+>
+> **2. A missing `VIEW DEFINITION` grant was reported as a missing proc.** This section has always specified
+> probe (a) as "`OBJECT_ID` of **both** procs", but the implementation folded (a) into (b) and inferred absence
+> from a NULL `OBJECT_DEFINITION`. **MEASURED** (2026-07-31, on the lab SQL Server): a principal holding only
+> `EXECUTE` on the proc gets a non-NULL `OBJECT_ID` and a **NULL** `OBJECT_DEFINITION`; the compat probe still
+> passes. So a deployed, working, correct procedure was reported as *missing*, and the operator was sent to fix
+> a `CREATE PROCEDURE` permission that was neither the cause nor the cure. `WITH ENCRYPTION` produces the
+> identical NULL and the identical misdiagnosis — and because *that* half needs no security principal, it is
+> now a live test leg (`test_a_deployed_proc_can_return_a_null_definition`), which pins on a real server the
+> one thing an offline stub cannot show: that the two functions genuinely disagree. The permission half stays
+> deferred with AC-10's other permission scenarios to a purpose-configured server.
+>
+> This is not a hypothetical posture here: §5's sub-lever B design explicitly serves "a fleet whose DB
+> principal can never hold `CREATE PROCEDURE`" — DBA-provisioned procs plus a least-privilege app principal —
+> which is exactly the deployment shape that hits it. The probe now returns `OBJECT_ID` beside the definition
+> and the two conditions get separate reasons:
+>
+> | condition | reason |
+> |---|---|
+> | `OBJECT_ID` NULL | genuinely absent — guarded DDL skipped, `CREATE PROCEDURE`/ALTER-on-schema denied, or a pre-2016-SP1 engine |
+> | `OBJECT_ID` non-NULL, `OBJECT_DEFINITION` NULL | deployed but unreadable — **`GRANT VIEW DEFINITION`**, or the module is `WITH ENCRYPTION` |
+>
+> Both still **degrade** — the gate hashes the body and cannot pass on one it cannot read — so no accept/reject
+> behaviour changed; only the diagnosis did. The probe SQL is pinned by an exact-match assertion in the
+> offline suite (a typo'd probe must fail loudly rather than silently match), so that pin moved with it and
+> stayed exact.
+
 **Versioning, mixed vintages, downgrade.** Procs are **name-versioned** (`_v1`, `_v2`, …): engine sharding runs
 N processes against ONE unified store (ADR 0037/0063), so a rolling upgrade briefly runs two builds against one
 database — each build calls exactly the body it shipped; a newer build's `_v2` never touches `_v1`. A retired
@@ -679,10 +738,18 @@ states, including the mismatch and 1222 translations). **Any miss = the flag sta
   injected-row test.
 - **AC-6** — The three flags SHALL be provable no-ops on SQLite and Postgres (neither backend references them).
   → sentinel test (the ADR 0075 precedent).
-- **AC-7** — WHEN `fifo_claim_proc` is ON and a proc is missing, its `OBJECT_DEFINITION` hash mismatches every
-  form this build deploys, or compat < 130, the store SHALL degrade loudly to the shipped batch (WARNING naming
-  the reason + degraded gauge), never a lane outage; the hot path SHALL contain no error-2812 handling. →
-  startup-gate tests incl. a hand-edited-body leg.
+- **AC-7** — WHEN `fifo_claim_proc` is ON and a proc is missing, is deployed but its definition unreadable
+  (`OBJECT_ID` resolves, `OBJECT_DEFINITION` NULL), its `OBJECT_DEFINITION` hash mismatches every form this
+  build deploys, or compat < 130, the store SHALL degrade loudly to the shipped batch (WARNING naming the
+  reason + degraded gauge), never a lane outage; the hot path SHALL contain no error-2812 handling. → startup-
+  gate tests incl. a hand-edited-body leg and an unreadable-definition leg.
+- **AC-7c** — The degraded gauge SHALL be a surface an operator can READ, not merely an attribute: `/status`
+  (with the reason string), `/metrics` (numeric, label-less, ABSENT rather than 0 when the lever is not
+  requested) and the console store panel SHALL each emit it. → surface-emission tests
+  (`test_adr0114_claim_proc_surfaces.py`), asserting the rendered output, not the property.
+  > Added by the 2026-07-31 amendment. AC-7 as written required a gauge and nothing required anyone to be able
+  > to see it; the two properties existed and were read by the store's own tests alone. A "loud" degrade whose
+  > only audience is a log line is how this lever stayed inert in every deployment for its whole life.
 - **AC-7b** — WHEN `fifo_claim_proc` is ON and both procs are deployed **by this build's own DDL**, the gate
   SHALL **PASS** and `claim_proc_effective` SHALL be True, verified against a **real SQL Server** (not a stub
   that echoes the submitted text back as the deployed body). → `test_adr0114_claim_proc_live.py`, plus an

messagefoundry/api/_ui_seam.py

@@ -75,7 +75,13 @@
 #: so an older console simply ignores it; bumped rather than corrected in place because v14 SHIPPED
 #: (v0.3.2). Under "one shipped posture, loosen only" a subset that reads as the whole posture is the
 #: failure this field exists to prevent, so the console must be able to render the caveat.
-ENGINE_UI_SEAM: int = 15
+#: seam v16: SystemStatus gained the additive `claim_proc` — ADR 0114 AC-7's degraded gauge (whether the
+#: SQL Server stored-procedure claim path passed its startup gate, and the reason string when it did
+#: not), which the status page's store panel renders. Additive with a default and `None` on every
+#: backend without the lever, so an older console simply ignores it; a separate seam rather than a
+#: correction to v15 because v15 is a SecurityPosture change and folding an unrelated DTO into it would
+#: make that note describe a field set it does not cover.
+ENGINE_UI_SEAM: int = 16
 
 
 @dataclass(frozen=True, slots=True)

messagefoundry/api/app.py

@@ -83,6 +83,7 @@
     CapturedResponseInfo,
     ChannelInfo,
     ClaimPoolInfo,
+    ClaimProcInfo,
     ClusterNode,
     ClusterNodeList,
     ClusterStatus,
@@ -4477,6 +4478,21 @@ async def system_status(
             if pool_status is not None
             else None
         )
+        # ADR 0114 AC-7's degraded gauge. Until this field existed the ONLY signal that the proc
+        # claim path had fallen back to the shipped batch was a WARNING at store open — which is a
+        # load-bearing part of why the gate could degrade in every deployment unnoticed. None unless
+        # [store].fifo_claim_proc is on and the backend has the lever, so the payload is unchanged
+        # by default. Synchronous + free (attributes the gate recorded once at open).
+        cps = engine.store.claim_proc_status()
+        claim_proc = (
+            ClaimProcInfo(
+                effective=cps.effective,
+                degraded_reason=cps.degraded_reason,
+                head_forms=dict(cps.head_forms),
+            )
+            if cps is not None
+            else None
+        )
         # App-log disk metering (#50), alongside the DB metrics — only when a log dir is configured.
         # Run the blocking stat()s off the event loop (the DB metering is itself off-loop in the store);
         # None when stdout-only or the directory is unreadable, so /status never raises on it.
@@ -4520,6 +4536,7 @@ async def system_status(
             logs=logs,
             update=update,
             pool=pool,
+            claim_proc=claim_proc,
         )
 
     # --- runtime log verbosity + redacted log-tail viewer (BACKLOG #171, ADR 0130) ----

messagefoundry/api/metrics.py

@@ -38,7 +38,12 @@
 
 from messagefoundry import __version__
 from messagefoundry.store.pool_metrics import PoolStatus
-from messagefoundry.store.store import DestinationMetrics, InboundMetrics, LatencyHistogram
+from messagefoundry.store.store import (
+    ClaimProcStatus,
+    DestinationMetrics,
+    InboundMetrics,
+    LatencyHistogram,
+)
 
 if TYPE_CHECKING:  # avoid pulling the heavy engine import into the default path
     from messagefoundry.pipeline import Engine
@@ -192,6 +197,10 @@
     pool: PoolStatus | None = None
     committed_txns: int = 0
     body_copies: int = 0
+    # ADR 0114 AC-7's degraded gauge. None when the backend has no fifo_claim_proc lever or the flag
+    # is off — the gauges are then ABSENT rather than 0, so a scrape can tell "not requested" from
+    # "requested and degraded" (a constant 0 on every SQLite fleet would be pure alert noise).
+    claim_proc: ClaimProcStatus | None = None
 
 
 async def gather_snapshot(engine: Engine) -> _Snapshot:
@@ -230,6 +239,7 @@
         pool=pool,
         committed_txns=committed_txns,
         body_copies=body_copies,
+        claim_proc=engine.store.claim_proc_status(),
     )
 
 
@@ -243,7 +253,7 @@
     def __init__(self, snap: _Snapshot) -> None:
         self._s = snap
 
     def collect(self) -> Iterable[Any]:
         s = self._s
 
         build = GaugeMetricFamily(
@@ -369,6 +379,33 @@
         body_copies.add_metric([], float(s.body_copies))
         yield body_copies
 
+        # ADR 0114 AC-7 degraded gauge. Emitted ONLY when [store].fifo_claim_proc is on: a constant
+        # 0 on every fleet that never asked for the lever is noise a scraper cannot alert on, and
+        # absence is the honest encoding of "not applicable here". Numeric and LABEL-LESS by
+        # design — the human-readable degrade reason is free text (it embeds a proc name and, on the
+        # probe-failure arm, an exception string), so carrying it as a label would both blow the
+        # cardinality budget and break this module's strict {connection,destination,status,version,le}
+        # allowlist. The reason string lives on /status and the console store panel instead.
+        cp = s.claim_proc
+        if cp is not None:
+            effective = GaugeMetricFamily(
+                "messagefoundry_store_claim_proc_effective",
+                "1 when the ADR 0114 stored-procedure claim path passed its startup gate and is"
+                " active, 0 when it degraded to the shipped ad-hoc batch (claims still flow).",
+            )
+            effective.add_metric([], 1.0 if cp.effective else 0.0)
+            yield effective
+            # Which stored head form the deployed modules matched. "verbatim" means this server did
+            # NOT rewrite the CREATE OR ALTER head — no engine measured to date does, so a fleet
+            # reporting 1 here is a live counterexample worth knowing about, not a fault.
+            verbatim = GaugeMetricFamily(
+                "messagefoundry_store_claim_proc_head_verbatim",
+                "1 when at least one deployed claim procedure's stored definition kept the CREATE"
+                " OR ALTER head verbatim (this server does not rewrite it), else 0.",
+            )
+            verbatim.add_metric([], 1.0 if "verbatim" in cp.head_forms.values() else 0.0)
+            yield verbatim
+
         # Connection-pool saturation + acquire-wait (server backends only; absent on SQLite, which has
         # no pool). [store].pool_size previously emitted NO saturation metric — these close that gap.
         pool = s.pool

messagefoundry/api/models.py

@@ -710,6 +710,26 @@ class PoolInfo(BaseModel):
     claim_pool: ClaimPoolInfo | None = None
 
 
+class ClaimProcInfo(BaseModel):
+    """The ADR 0114 sub-lever A (``fifo_claim_proc``) startup-gate verdict — AC-7's **degraded
+    gauge**, surfaced as the additive ``claim_proc`` field on :class:`SystemStatus`.
+
+    ``None`` on every backend without the lever and on SQL Server when the flag is off, so "not
+    requested" reads differently from "requested and degraded". When ``effective`` is False,
+    ``degraded_reason`` says why the store fell back to the shipped ad-hoc batch — claims keep
+    flowing either way, so this is a performance-lever gauge, not a health alarm.
+
+    Metadata only: proc names, a head-form word, and the gate's own reason string — no message
+    content and no PHI."""
+
+    effective: bool  # the gate passed; pooled claims run through the procs
+    degraded_reason: str | None = None  # why it degraded to the batch; None when effective
+    # proc name -> the stored head form the deployed module matched ("rewritten" | "verbatim").
+    # "verbatim" means this server does NOT rewrite CREATE OR ALTER — no engine measured to date
+    # does, so it is worth reporting; it is an engine difference, not a fault.
+    head_forms: dict[str, str] = Field(default_factory=dict)
+
+
 class SystemStatus(BaseModel):
     engine: EngineInfo
     # Engine-wide top-line roll-up KPIs (#93): total messages, combined in+out connection count with
@@ -727,6 +747,10 @@ class SystemStatus(BaseModel):
     # percentiles + size/idle occupancy). Additive + ``None`` on SQLite (no pool) so the existing
     # payload is unchanged on the default backend and an older client deserializes /status unchanged.
     pool: PoolInfo | None = None
+    # ADR 0114 AC-7's degraded gauge: whether the SQL Server proc claim path is effectively active,
+    # and why not when it isn't. Additive + ``None`` on every backend without the lever and whenever
+    # [store].fifo_claim_proc is off, so the default payload is unchanged.
+    claim_proc: ClaimProcInfo | None = None
 
 
 class IntegrityResult(BaseModel):

messagefoundry/store/base.py

@@ -48,6 +48,7 @@
     AlertInstance,
     CapturedResponse,
     ClaimedHeads,
+    ClaimProcStatus,
     ConnectionEvent,
     ConnectionMetrics,
     DbStatus,
@@ -1294,6 +1295,17 @@ def pool_status(self) -> PoolStatus | None:
         on SQLite (no pool)."""
         ...
 
+    def claim_proc_status(self) -> ClaimProcStatus | None:
+        """The ADR 0114 sub-lever A stored-procedure-claim startup-gate verdict, or ``None`` when
+        this backend has no such lever (AC-6: SQL Server is the only one that reads its flag, whose
+        literal name this module therefore does not write) or that flag is off. AC-7's **degraded
+        gauge** — the surface an operator can actually see the degraded
+        state on (``/status``, ``/metrics``, the console store panel); before it existed the whole
+        signal was one WARNING at ``open()``. Synchronous + free (three attributes the gate recorded
+        once at open — no DB round-trip), read-only, and additive: the ``/status`` field defaults
+        ``None``, so an older client deserializes it unchanged."""
+        ...
+
     async def integrity_check(self) -> tuple[bool, str]: ...
 
     async def connection_metrics(

messagefoundry/store/postgres.py

@@ -108,6 +108,7 @@
     AlertInstance,
     CapturedResponse,
     ClaimedHeads,
+    ClaimProcStatus,
     ConnectionEvent,
     ConnectionMetrics,
     DbStatus,
@@ -1274,6 +1275,11 @@
             acquire_wait=self._acquire_wait.summary(),
         )
 
+    def claim_proc_status(self) -> ClaimProcStatus | None:
+        """``None``: the ADR 0114 sub-lever A stored-procedure claim path is SQL-Server-only (AC-6 —
+        this backend never reads its flag), so there is no gate verdict to report here."""
+        return None
+
     async def _fetchall(self, sql: str, *params: Any) -> list[Any]:
         return list(await self._pool.fetch(sql, *params))