fix(llm-rubric): decline instead of fabricating on cross-artifact predicates (closes #291)

[uda-lab-agent]

What & why

Closes #291 (dogfood finding on trunk #63 — semantic rubric quality).

When an llm-rubric rule's predicate ranges over artifacts not present in the rendered target (e.g. "the scaffold-status table should classify every module under src/", judged against only the doc), prompt v7 emitted a confident pass/fail with fabricated within-target grounding instead of declining. In the #291 live run against yomotsusaka's repo-rules.md (gpt-5.4), this:

• missed the real drift — 13 modules absent from the table (the dominant, actual violation) were never mentioned, because the backend never sees the src/ tree;
• false-positived on the reason it did give — flagged a legitimately deferred stub (transfer.py) as "already implemented";
• false-failed two source-code-invariant rules (L163, L164) judged against a doc target they cannot ground.

Fix — direction (a): decline rather than fabricate

Extends the existing unsupported decline contract. Before v8 the only authorised decline was the target-kind-mismatch case (gated on a target_kind annotation). v8 adds a second reason discriminator:

"unsupported_reason": "target_kind_mismatch" | "cross_artifact_predicate" | null

cross_artifact_predicate is valid for any rule (annotated or not): the model returns unsupported and names the absent artifacts in primary_reason instead of guessing. The backend maps it to Status.UNSUPPORTED with evidence.kind=cross_artifact_predicate and a remediation pointing at the existing S5 remedy — params.target_scope / multi-target (PR #290, docs/design/multi-target.md §9). Direction (b) already shipped; this fix points authors at it rather than rebuilding it.

Fail-closed: UNSUPPORTED is a non-pass status — an explicit, evidence-bearing non-verdict, never a silent pass.

Over-trigger guard: the instruction fires only when the absent artifacts are essential to the predicate. A rule that merely names another file but is decidable from the target itself is judged normally. Encoded in the prompt wording and tested on both sides.

Aggregation (following the target-kind-mismatch precedent): a cross-artifact decline counts as an ordinary unsupported vote under consensus, and is conclusive without escalation under review / adaptive (a re-run against the same insufficient target cannot recover it).

PROMPT_VERSION v7 → v8 (the eval-cache keys on it by design, so old entries correctly miss).

Before / after (live, gpt-5.4)

Before (v7, #291 verdict against the doc): status: fail with a fabricated reason ("transfer.py … already implements its MVP responsibility") — wrong, and silent about the 13 missing modules.

After (v8, #291-shaped bench fixture): status: unsupported

Determining whether the scaffold-status table covers every module under src/pkg/ excluding __init__.py requires seeing the actual src/pkg/ source tree, which is not included above.

Validation

• uv run pytest — 1751 passed (7 new TestCrossArtifactPredicateDecline tests + updated v7→v8 assertions).
• uv run ruff check ., uv run ruff format --check ., uv run pyright — all green.
• textlint on changed docs — exit 0.
• Live bench (gpt-5.4, prompt v8), 5-entry slice, reproducibility=2: 5/5 correct, avg reproducibility 1.00.
  • completeness-06-cross-artifact-src-tree → unsupported (new decline reproduces on the real model)
  • completeness-07-self-contained-doc-header → pass (over-trigger guard holds on the same target)
  • clarity-01 → pass, clarity-02 → fail (no regression on grounded rules)
  • target-kind-mismatch-01 → unsupported (existing decline path intact)
  • API cost: ~12k input / ~800 output tokens across ~11 calls (single-digit cents).

Files

• src/gate_keeper/backends/llm_rubric.py — schema field + parser, both prompt templates (v8), decline helpers, single/consensus/review wiring, adaptive docstring.
• src/gate_keeper/bench.py — recognise cross_artifact_predicate evidence for expected-unsupported entries.
• tests/fixtures/semantic/ — two llm-rubric: completeness/cross-artifact rules get fabricated evidence + missed drift (dogfood, #63) #291-modeled fixtures + target.
• docs/semantic-rules.md, docs/llm-rubric.md — cross-artifact-predicate guidance + target_scope remedy.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wx8qUs1ipy6Nf82hu2eDhg

[t-uda]

@codex please review this PR

[github-actions]

gate-keeper dogfooding

rule	status	judgment	primary_reason
rule-dogfooding-rules-L23	PASS	pass	The PR description body clearly explains the user-visible behavioral change: the system now declines unsupported cross-artifact rubric judgments instead of fabricating grounded pass/fail verdicts.
rule-dogfooding-rules-L24	PASS	pass	The PR description includes an explicit validation section with test commands, results, and live verification details.
rule-dogfooding-rules-L25	PASS	pass	The PR description does not hardcode any absolute filesystem locations tied to a specific user, username, or workspace root.

Tokens: in=9546 out=615, latency: 7306 ms, model: gpt-5.4, prompt_version: v8

_{See docs/dogfooding.md for the per-rule promotion path.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 948d4c25f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".