Proposal: graded confidence — an epistemic-reliability axis, distinct from integrity (#140) and citation (#92/#94)

[K4uP]

Following up on my comment in #95, after reading how the trust surface developed
this week (#140 cryptographic provenance, #92/#94 SOURCES + inline citation, the
new SPEC §12 trust-and-safety in #58). Those cover three distinct senses of
"trust" — and there's a fourth still open, which is the one we've had to solve in
production.

Four axes, so we don't talk past each other:

Axis	"Trust that…"	Where it lives now
Authenticity / integrity	the bundle is from who it claims, unaltered	#140 (signing, SHA-256 manifest)
Safety	content is data, not instructions (prompt-injection)	#58 §12
Groundedness / citation	a claim is linked to evidence	#92 SOURCES, #94 inline citation
Reliability (this proposal)	how much to believe the claim itself	— open —

A bundle can be perfectly signed (#140), injection-safe (#58), and have every
claim cited (#94) — and an agent still can't tell a claim verified against the
live system from one that merely restates vendor documentation (often stale or
aspirational) from one that's inferred and unverified. A citation proves a claim
is grounded; it doesn't grade the ground. For a retrieval agent that's tolerable.
For an agent that acts on a real system, it's where damage happens. (This is
the dimension #95 explicitly scoped out.)

What we run in production — single-domain agent KB, enterprise IT service
management, ~250 atomic notes over ~140 build cycles, agents acting on live
customer systems:

• a graded confidence value per concept in frontmatter —
  HIGH | MEDIUM | LOW | UNVERIFIED — orthogonal to "is it cited";
• a small evidence-kind vocabulary on claims (observed-on-system /
  documentation / partner-confirmed / inferred) — this types the kind of
  evidence, not just its URL, so it's a different sense of "provenance" than
  Provenance / verification: an additive layer for OKF bundles #140's
  origin provenance;
• an explicit trust ordering when two cited sources disagree
  (observed > partner-confirmed > observed+doc > doc > inferred) and a
  conflict rule: document both sides, never silently pick.

The taxonomy is domain-specific; the axis is not. Every producer whose agents
act reinvents some version of "how sure are we", and without a shared optional
slot none of it survives the cross-org hop OKF exists for.

The ask — minimal, optional, already spec-permitted ("producers MAY include
additional keys"):

• an optional confidence: frontmatter key with a small recommended enum, and
• an optional convention for typing evidence-kind on inline citations (riding on
  Proposal: Inline citation as default convention for coupling claims to evidence #94,
  complementary to Provenance / verification: an additive layer for OKF bundles #140
  — different sense of "provenance").

Not a mandate; minimal producers ignore it. I'm not attached to the names — happy
to align with #140/#92 so we don't end up with three meanings of "provenance".

Questions:

1. Is graded epistemic confidence in scope as an optional OKF convention, or
   deliberately producer-side forever?
2. If in scope — its own key, or folded into the
   Proposal: SOURCES body section for agent navigation and provenance #92 SOURCES /
   Provenance / verification: an additive layer for OKF bundles #140 provenance work?
3. Would a worked example bundle (an OKF-shaped slice of our corpus with the
   confidence layer applied) help the discussion?

Thanks for the pace and openness here — the trust model is clearly being built
right now, which is why I wanted to get this axis on the table before it settles.

[Dynamicfeedai]

Strong framing. Mapping to where I have a running implementation: integrity (#140) answers "who produced this, unaltered" (a signature, C2PA). Citation (#92/#94) answers "what the claim's source is." This axis (#151) answers the orthogonal question: how much should a consumer believe the claim, even when it is perfectly signed and correctly cited.

A perfectly signed, correctly cited claim can still be measured from the live system, restated from vendor docs, or inferred, and those are very differently trustworthy. That is the gap this axis fills.

Here is a concrete strawman for a per-claim reliability object, grounded in a shape we run live today (fact/v1), so it is battle-tested rather than theoretical:

"reliability": {
  "confidence": 0.8,
  "basis": "live-source",
  "sources": 1,
  "verified": false,
  "freshness": { "as_of": "2026-06-28T03:00:00Z", "expires": "2026-06-28T04:00:00Z", "state": "live" },
  "signals": { "signed": true, "corroborated": false, "fresh": true }
}

Field intent:

• confidence (0..1): a transparent composite of the signals below, never decorative. A consumer should be able to recompute it.
• basis: how the claim was obtained: live-source (measured or reported by an authoritative source), vendor-doc (restating a vendor's published value), inferred (model or heuristic), computed (derived). The measured vs restated vs inferred distinction, made explicit and machine-readable.
• sources: count of independent sources behind the claim (complements the citation axis, which names them).
• verified: true ONLY when corroborated by 2+ independent sources AND fresh. Crucially, signed is not verified: a single-source claim is verified: false even when perfectly signed. Integrity is axis Provenance / verification: an additive layer for OKF bundles #140, not this one.
• freshness: as_of / expires / state, so a stale claim is visible rather than silently trusted.
• signals: the booleans confidence is composed from, exposed for auditability.

Two rules that keep it honest:

1. verified never piggybacks on the signature. Signing proves integrity (Provenance / verification: an additive layer for OKF bundles #140); graded reliability requires independent corroboration.
2. confidence is a function of the exposed signals, so a reviewer can recompute it. No opaque scores.

We run exactly this on every datapoint today (verification block: confidence / sources / verified / basis / signals). Happy to contribute a JSON Schema file plus conformance examples for this object, and to keep the naming clean so #140 owns integrity, #92/#94 own citation, and #151 owns graded reliability.

[K4uP]

This is exactly the convergence I hoped for — you're running in production what we arrived
at independently, which is the best signal the axis is real. Strong +1 on the reliability
object and on both honesty rules (signed ≠ verified; confidence recomputable from exposed
signals). The shape, the basis enum, and handling an ordinal vs. a computed score sensibly
all line up with us — so rather than re-litigate those, three refinements that come specifically
from a human-authored, multi-version corpus and that a live-data layer doesn't surface:

1. The state the strawman is missing: conflict, not just absence of corroboration.
You capture sources and corroborated, and reserve verified for 2+ agreeing — good. But
that assumes sources either corroborate or stay silent. In a real corpus they sometimes
disagree (the vendor doc says X, the live system does Y). Collapsing that into
corroborated:false loses the most important signal. We carry an explicit conflict state —
rule: "document both sides, never silently pick" — plus a trust ordering to act under it
(live > partner > live+doc > doc > inferred). The object needs a disputed signal distinct from
"merely uncorroborated"; an acting agent should behave very differently in each.

2. Two different clocks: freshness ≠ validity.
Your freshness (as_of/expires, your example a ~1h TTL) is measurement recency — right for a
live metric. A versioned-knowledge corpus has a second, unrelated time axis: applicability —
a fact holds for product release range X–Y, independent of when it was measured (the
valid_from/valid_until thread on #87 / KCP). And validity is often keyed on version, not
wall-clock, so a timestamp expires can't express it. I'd keep freshness (recency/TTL) and
a separate validity (applicability window, version- or time-keyed) as two fields — conflating
them breaks one use case or the other.

3. confidence is a lifecycle, not a write-once value.
The thing our import pipeline made unavoidable: a claim's reliability changes over its life.
New information enters provisional and low, then earns its grade through verification — we run
an explicit promotion ladder (IMPORT-RAW (LOW) → IMPORT-VERIFIED (MEDIUM) → live-confirmed (HIGH)),
and it runs in reverse too: a HIGH fact silently goes stale when the platform version
moves, gets flagged, and is demoted until re-verified. Note this is the opposite direction
from your freshness decay — yours handles trust expiring, this handles trust being earned;
the axis needs both. Two implications: (a) a "last graded at" timestamp distinct from
freshness.as_of — when measured and when last assessed are different events; (b) basis
and confidence co-evolve (often inferred/LOW on intake → live-source/HIGH once verified),
so they update together, ideally with a small audit trail of transitions. This extends your
auditability rule from "recomputable at a point in time" to "recomputable across its lifetime."

Strong yes to the naming split (provenance = integrity/#140, reliability/assurance = #151) —
that kills the three-meanings-of-"provenance" problem directly. And yes please to the JSON Schema +
conformance examples — happy to co-author, and to contribute a worked example bundle from our
multi-version corpus so the schema gets tested against both a live-data shape (yours) and a
human-authored, version-scoped one (ours).

[K4uP]

I took your strawman plus the three refinements above and sketched a concrete reliability object — sharing it here for critique before anyone touches a PR. It keeps your shape and both rules (signed != verified; recomputable, not opaque) and folds in the conflict / validity / lifecycle points.

A worked instance — the conflict case, which the strawman couldn't yet express:

reliability:
  confidence: MEDIUM          # ordinal band = the interoperable surface
  basis: live-source          # (+ partner-attested added to your enum)
  sources: 2
  verified: false             # sources disagree -> not corroborated
  conflict:
    disputed: true
    positions:                # both retained, never silently pick
      - { statement: "Max attachment 100 MB", basis: vendor-doc,  source: "Admin Guide 5.0 p.142" }
      - { statement: "Max attachment ~50 MB", basis: live-source, source: "Observed on a 5.0 instance; 60 MB rejected" }
    resolution: "live-source prevails under the trust ordering; both positions kept"
  validity:  { keyed_by: version, valid_from: "5.0", valid_until: null }   # version-keyed, not wall-clock
  freshness: { as_of: "2026-06-18T00:00:00Z", state: fresh }               # a separate clock
  signals:   { signed: true, corroborated: false, fresh: true }

And the lifecycle, showing a grade is earned, not stamped once:

  assessed_at: "2026-06-20T09:30:00Z"   # when graded -- distinct from freshness.as_of (when measured)
  lifecycle:
    stage: live-confirmed
    history:
      - { at: "2026-05-02", confidence: LOW,    basis: inferred,    note: "import intake, unverified" }
      - { at: "2026-05-09", confidence: MEDIUM, basis: vendor-doc,  note: "matched vendor docs" }
      - { at: "2026-06-20", confidence: HIGH,   basis: live-source, note: "observed live + corroborated" }

Deltas from your strawman, all additive:

• confidence is an ordinal band; score is optional. A 0..1 float is honest only when computed — human-authored grades can't distinguish 0.7 from 0.8 and get thresholded back into bands anyway. So the band is the field everyone filters on; score (+ your signals) rides along only when actually computed. Your "no opaque scores" rule holds: an absent score means "not computed", not "hidden".
• conflict object. corroborated: false can't distinguish "no second source yet" from "two sources contradict". This makes disagreement first-class — both positions retained, resolution records what the trust ordering picked (live > partner > vendor-doc > inferred) without discarding the loser.
• freshness vs validity are two fields. Yours (as_of/expires) is measurement recency. validity is the applicability window — often version-keyed (5.0..null), which a timestamp expires can't express. This is also where KCP's production experience with OKF's three open problems — and a bridge command #87's valid_until lands.
• basis += partner-attested (an independent integrator/partner; above vendor-doc, below live).

I've written this up as a full JSON Schema (draft 2020-12) plus a small, fully synthetic example bundle (two concept notes — the conflict and lifecycle cases). Happy to share both however's most useful: a gist, a follow-up comment here, or a PR if you'd rather have them in-repo.

Two things I'd especially value your read on:

1. does the conflict object belong on this axis, or is "document both sides" a separate concern?
2. OK to make ordinal confidence the required field and score the optional companion?