feat(llm-rubric): narrowed affected-context evaluation (closes #281) - #290
Merged
Conversation
Generalize the multi-target prompt-assembly path so an engine-computed dynamic file set — (S1 changed set) ∩ (S3 rule scope) — routed to llm-rubric is assembled into a single narrowed prompt with per-file headers, under a token budget, with deterministic lexicographic truncation and auditable evidence. Replaces multi_target_unsupported on this path only; the literal params.targets (≤5) mechanism and the github/external backends keep the unchanged decline. Implements S5 of umbrella #277 (incremental audit trunk), per docs/design/multi-target.md §2.2-2.3 and the new §9.12. - _assemble_affected_context / _run_affected_context in llm_rubric.py: read the effective set, order lexicographically, concatenate under `--- <path> ---` headers, evaluate in one provider call per strategy iteration via the existing single-evaluation machinery. - Token budget precedence params.token_budget > dotenv GATE_KEEPER_TOKEN_BUDGET > 32000; estimate ceil(len/3.2). Over-budget truncates to a lexicographic prefix and names omitted files in truncation_warning; zero survivors fail closed to UNAVAILABLE with affected_context_empty (no provider call). - eval-cache (#69 S2-B): the assembled-content hash is the target content hash — _build_target_entries emits one __affected_context__ entry hashing the surviving assembled body, so budget-driven truncation is reflected in the key (criterion 4). - Mixed target_kind (criterion 5 / §9.7): decided in §9.12.3 — the dynamic path assembles every in-scope file without per-file kind gating; the run-level #178 precheck stays the only kind gate. - Docs: multi-target.md §9.12 + status pointers, eval-cache.md §9, rule-ir.md token_budget row, cli-reference.md. - Tests: tests/test_affected_context.py covers all six acceptance criteria incl. a fixture demonstrating whole-closure vs narrowed token estimate; updated stale multi_target_unsupported assertions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wx8qUs1ipy6Nf82hu2eDhg
Owner
|
@codex please review this PR |
gate-keeper dogfooding
Tokens: in=7707 out=624, latency: 6965 ms, model: gpt-5.4, prompt_version: v7 See |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wx8qUs1ipy6Nf82hu2eDhg
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9334bbfedb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…x P2 #290) Codex P2: the S5 affected-context cache entry hashed only the included assembled body, so a file that stays omitted under the same budget — when added or edited — left the key unchanged and could hit an old entry, serving stale affected_context / truncation_warning evidence that never named it. The omitted and unreadable files, and the effective token budget, surface in the cached diagnostic's evidence but never enter the assembled body, so they must participate in the key. The dynamic targets entry now also carries the omitted files ({path, content_sha256} each), the unreadable labels, and token_budget. _AssemblyResult exposes omitted_files (with content) alongside the omitted-label list. Any change to the omitted / unreadable set, an omitted file's content, or the budget (incl. a dotenv GATE_KEEPER_TOKEN_BUDGET change that leaves survivors unchanged) now misses and recomputes correct evidence. Tests: test_omitted_file_change_changes_key, test_budget_change_without_truncation_change_changes_key. Docs: multi-target.md §9.12.4 and eval-cache.md §9 updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wx8qUs1ipy6Nf82hu2eDhg
This was referenced Jul 5, 2026
uda-lab-agent
added a commit
that referenced
this pull request
Jul 5, 2026
…dicates (closes #291) (#292) * fix(llm-rubric): decline instead of fabricating on cross-artifact predicates (closes #291) When a rule's predicate ranges over artifacts absent from the rendered target (e.g. "classify every module under src/" judged against only the doc), prompt v7 emitted a confident pass/fail with fabricated within-target grounding rather than declining. The #291 dogfood run missed real drift (13 unclassified modules), false-flagged a correct `deferred` classification, and false-failed two code-invariant rules against a doc target. Fix direction (a): broaden the decline contract. v8 adds a second authorised `unsupported` reason, `unsupported_reason: "cross_artifact_predicate"`, valid for any rule (annotated or not). The model returns `unsupported` and names the absent artifacts instead of guessing; the backend maps this to `Status.UNSUPPORTED` with `evidence.kind=cross_artifact_predicate` and a remediation pointing at the existing S5 remedy (`params.target_scope` / multi-target, PR #290 / multi-target.md §9). Fail-closed: an explicit, evidence-bearing non-verdict, never a silent pass. Over-trigger guard: the instruction fires only when the absent artifacts are essential to the predicate — a rule that merely names another file but is decidable from the target is judged normally. Self-contained rules keep judging pass/fail. Aggregation (following the target-kind-mismatch precedent): a cross-artifact decline counts as an ordinary `unsupported` vote under consensus, and is conclusive without escalation under review/adaptive (a re-run against the same insufficient target cannot recover it). PROMPT_VERSION v7 -> v8 (eval-cache keys on it by design; old entries miss). Tests: parser accepts the new shape; single-strategy maps to UNSUPPORTED for annotated and unannotated rules; self-contained pass is not declined; consensus counts the decline as an unsupported vote; stray reason on pass is dropped. Two bench fixtures model the #291 scenario (a doc table + a src-tree completeness rule -> unsupported; a self-contained companion -> pass). Docs: docs/semantic-rules.md (cross-artifact predicates) and docs/llm-rubric.md (decline verdicts) document the behavior and the target_scope remedy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wx8qUs1ipy6Nf82hu2eDhg * fix(eval/bench): count cross_artifact_predicate evidence as telemetry-bearing Codex P2 (PR #292): a bench entry expecting `unsupported` that receives the new `cross_artifact_predicate` decline evidence was marked PASS, but the telemetry loop only recognised `llm_judgment` / `target_kind_mismatch` / `llm_quote_fabrication`, so those #291 fixtures reported tokens_in/out=0, latency_ms=0, and model/prompt_version=None despite the evidence carrying those fields. Add `cross_artifact_predicate` to `_TELEMETRY_BEARING_KINDS` and cover it with a bench test mirroring the target-kind-mismatch telemetry test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wx8qUs1ipy6Nf82hu2eDhg --------- Co-authored-by: uda-lab-agent <uda-lab-agent@hermes-engineering.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements S5 of umbrella #277 (incremental audit trunk): closes #281.
A multi-file
TargetSpecrouted tollm-rubric— the narrowed affected context,(S1 changed set) ∩ (S3 rule scope), never the whole reference closure — is now assembled into a single prompt with per-file--- <path> ---headers, evaluated in one provider call per strategy iteration, under a token budget with deterministic lexicographic truncation and auditable evidence.multi_target_unsupportedis replaced on this path only; the literalparams.targets(≤5 entries) mechanism and thegithub/externalbackends keep the unchanged decline.Design:
docs/design/multi-target.md§2.2–2.3 and the new §9.12 (ratified S5 contract).Per-criterion evidence
_run_affected_contextassembles and dispatches through the existing single-evaluation machinery (one provider call × strategy N).affected_contextevidence listsincluded_files+tokens_estimated. Tests:TestCriterion1SingleCallPerFileHeaders(direct spec + full scope/changed path).params.token_budget> dotenvGATE_KEEPER_TOKEN_BUDGET>32000; estimateceil(len/3.2). Truncation keeps a lexicographic prefix; dropped files are named intruncation_warning. Tests:TestCriterion2Truncation.affected_context_emptyevidence; covers both over-budget lead file and all-unreadable sets. Tests:TestCriterion3ZeroSurvivors.eval_cache._build_target_entriesemits one__affected_context__entry whosecontent_sha256hashes the surviving assembled body (re-running the same deterministic assembler), so budget-driven truncation is reflected in the key. Tests:TestCriterion4CacheKey(incl. content-edit miss, truncation-change miss, integration hit → zero extra calls).target_kind— decided in §9.12.3. The dynamic path assembles every in-scope file without per-file kind gating (per-file kind inference is the heuristic §2.2 forbids; silent drops break the "no file without a truncation_warning" invariant). The run-level Deterministically short-circuit target_kind mismatch before invoking llm-rubric #178 precheck stays the only kind gate, unchanged and run-level (§9.7). Documented in the same PR (multi-target.md§9.12.3, §9.7). Tests:TestCriterion5MixedKind.TestCriterion6WholeVsNarrowedruns the same scoped rule against the full closure and a narrowed changed set, asserting the narrowedtokens_estimatedis materially lower.Decisions worth noting
multi-target.md§9.12.3.params.targetspreserved: a rule declaring the literalparams.targetslist handed a multi-fileTargetSpeckeepsmulti_target_unsupported— the two axes never mix on one code path.Verification
uv run pytest(1742 passed),uv run ruff check .,uv run ruff format --check .,uv run pyright(0 errors) all green. textlint clean on changed docs.Docs
multi-target.md§9.12 + status pointers (§2.2, §9.7, §9.10),eval-cache.md§9,rule-ir.mdtoken_budgetrow,cli-reference.mdnew "Narrowed affected-context assembly" section.Closes #281. Part of umbrella #277.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Wx8qUs1ipy6Nf82hu2eDhg