Skip to content

feat(orb): content-addressed decision records on every verdict - #8842

Merged
JSONbored merged 1 commit into
mainfrom
feat/decision-record
Jul 26, 2026
Merged

feat(orb): content-addressed decision records on every verdict#8842
JSONbored merged 1 commit into
mainfrom
feat/decision-record

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

What

Every terminal verdict now persists a content-addressed decision record, and the public review surface renders it. Closes #8836 (epic #8828, Phase 4 — the highest impact/cost item of the trust-surface research). Stacked on #8840 (migration contiguity: 0179 follows its 0178).

Why

A contributor closed by ORB today cannot see which ruleset version or clause closed them — and the authoritative config being private (the operator's config dir overrides the public yml) makes that worse. The remedy from the epic's research: publish the digest even where contents stay private. "The bot closed me" becomes "clause guardrail_hold of config abc123… decided this" — inspectable, arguable, and stable under challenge. The shape follows SLSA's Verification Summary Attestation, which exists for exactly this delegated-decision pattern.

This record is also the input schema the golden-corpus replay (#8832) and the deterministic replay harness (#8838) consume — one schema, three consumers, so "what we published" and "what we can replay" can never drift apart.

How

  • canonicalJson — recursively key-sorted, whitespace-free, the ONE serialization every digest is computed over. Key order is a construction artifact, never meaning; un-JSON-able values throw loudly (a silent wrong digest is worse). Property-tested: key-order invariance at depth, array-order preservation, unicode stability.
  • DecisionRecord — action (the acted disposition, never the raw conclusion — gate_decision can contradict the action taken — stale 'merge' verdicts recorded after CI-failure closes distort calibration #8825's lesson), the clause (reasonCode, same expression as recordNativeGateDecision at the same site, so record and calibration row can never disagree about why), resolved-config digest, gate pack, and model/prompt commitments (null for rule-only decisions; AI-review wiring is the tracked follow-up).
  • Persistence (migration 0179): latest-finalize-wins per (target, head sha), best-effort — legibility must never break finalization. The stored JSON is the canonical form: re-digesting the stored bytes reproduces the stored digest (tested — that is the replay check).
  • Public surface: the unified comment appends a bounded "Decision record" collapsible from the stored record on republish. First publish precedes first finalize by design; the section rides the next republish and a read failure just omits it.

Verification

  • TSC clean; 550 tests green across decision-record/queue/queue-2/lifecycle/migration-collisions, including an end-to-end republish test asserting the collapsible, the clause, and the truncated digest appear in the posted comment body.
  • Changed-line coverage: 0 uncovered statements/branches. Branding + schema-drift checks green.

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

Base automatically changed from feat/decision-audit-sampling to main July 26, 2026 11:51
@JSONbored JSONbored self-assigned this Jul 26, 2026
A contributor closed by ORB cannot see WHICH ruleset version or clause
closed them -- and the authoritative config being private makes that
worse, not better. Every terminal disposition now persists a canonical,
content-addressed record and the public review surface renders it:
'the bot closed me' becomes 'clause X of config abc123 closed me' --
inspectable, arguable, stable under challenge. Shape follows SLSA's
Verification Summary Attestation.

Closes #8836. Part of epic #8828 (Phase 4 -- trust surface). Stacked
on the #8830 branch (migration contiguity with 0178).

- decision-record.ts: canonicalJson (recursively key-sorted -- key
  order is construction artifact, never meaning; refuses un-JSON-able
  values loudly), sha256Hex via Web Crypto, buildDecisionRecord
  (normalizes optional-shaped fields once, at the builder),
  persistDecisionRecord (latest-finalize-wins per target@sha,
  best-effort -- legibility never breaks finalization),
  renderDecisionRecordSection (bounded), loadDecisionRecordCollapsible
  (fail-safe null)
- migration 0179: decision_records
- processors: persist at the SAME finalize site as
  recordNativeGateDecision with the SAME reasonCode expression, so the
  record and the calibration row can never disagree about WHY; the
  unified comment appends the stored record as a collapsible on
  republish (first publish precedes first finalize by design)
- model/prompt commitments are null for rule-only decisions; wiring
  the AI-review contribution is the tracked follow-up on #8836
@JSONbored
JSONbored force-pushed the feat/decision-record branch from dd85f5a to f237a05 Compare July 26, 2026 11:54
@JSONbored
JSONbored merged commit 9537fef into main Jul 26, 2026
2 checks passed
@JSONbored
JSONbored deleted the feat/decision-record branch July 26, 2026 11:57
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
21829 2 21827 21
View the top 2 failed test(s) by shortest run time
test/unit/config-templates.test.ts > config/examples review templates (#1682) > loopover.full.yml body matches .loopover.yml.example from WHERE IT LIVES onward
Stack Traces | 0.0313s run time
AssertionError: expected '# WHERE IT LIVES (first match wins):\…' to be '# WHERE IT LIVES (first match wins):\…' // Object.is equality

- Expected
+ Received

@@ -91,13 +91,10 @@
  #   off      — the dimension is not evaluated.
  #   advisory — the finding is surfaced (comment/context) but never blocks.
  #   block    — the finding can become a hard `LoopOver Gate` blocker
  #              (always confirmed-contributor-gated).
  gate:
-   # Percent (0-20) of would-auto-close PRs randomly HELD for human adjudication instead of closing --
-   # the randomized holdout behind the unbiased close-precision estimate (#8831). 0/absent disables.
-   # closeAuditHoldoutPct: 5
    # Legacy check-run publish switch (#5355) — despite the name, this does NOT turn the deterministic
    # gate itself on or off. Gate evaluation, comments, labels, audit records, spend, and autonomous
    # merge/close all run identically whether this is true, false, or unset; the per-dimension modes
    # below are what actually configure gate evaluation. `enabled` is only a boolean shorthand for
    # `checkMode` below: true maps to `required`, false maps to `disabled`, and it cannot express

 ❯ test/unit/config-templates.test.ts:39:57
test/unit/selfhost-metrics.test.ts > DEFAULT_METRIC_META completeness (drift guard, 2026-07 fix) > every literal metric name emitted anywhere in src/ has a registered DEFAULT_METRIC_META entry
Stack Traces | 0.216s run time
AssertionError: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "loopover_close_audit_holdouts_total",
+ ]

 ❯ test/unit/selfhost-metrics.test.ts:385:21

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

JSONbored added a commit that referenced this pull request Jul 26, 2026
… in the decision record (#8847)

When an AI-judgment finding (ai_consensus_defect / ai_review_split)
shaped a decision, the decision record now carries the finding's
calibrated confidence and a digest of the prompt template commitment
(REVIEW_PROMPT_VERSION + REVIEW_SYSTEM_PROMPT) -- the fields that join
every decision to the risk-control calibration set (#8835) and complete
#8842's tracked follow-up.

Advances #8834 (the persistence half its issue text names as the core
deliverable; the flag-gated rotated-exemplar N-run variant remains --
the existing dual-model consensus already IS a two-sample agreement
signal, so the extra runs are a cost knob, not the prerequisite).

- DecisionRecord schema v2: + aiConfidence (null when no AI judgment
  contributed); normalized once at the builder like its siblings
- processors: source the confidence from the gate's own AI-judgment
  blocker; modelId deliberately stays null at this site -- the finding
  does not carry which concrete models ran, and a guess would be worse
  than nothing (reviewDiagnostics holds per-run identities)
- render: the model line gains the confidence when present
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

trust: content-addressed decision record (SLSA-VSA shape) published with every verdict

1 participant