diff --git a/docs/BEST_USE_CASE_2026_06_27.md b/docs/BEST_USE_CASE_2026_06_27.md new file mode 100644 index 0000000..8ae470f --- /dev/null +++ b/docs/BEST_USE_CASE_2026_06_27.md @@ -0,0 +1,188 @@ +# Dorian's Best Use Case — 27 June 2026 + +> An evidence-backed answer to "what is the single highest-leverage thing to use Dorian for today?" +> Built from a multi-agent web-research pass (37 cited sources), real-repo trials on the published +> `dorian-vwp` 1.2.0, a competitive analysis, adversarial critique, and an independent judge. Follows +> the honesty contract in [`VALIDATION_HONESTY.md`](VALIDATION_HONESTY.md). + +## Executive answer + +**Use Dorian as the *receipt* for the no-test facts an AI coding agent claims it changed — and the +*alarm* when a later commit silently makes one of them false.** + +When Claude Code / Codex / Cursor finishes a change, its summary is full of specific assertions — +"the signature is unchanged," "`requires-python` is `>=3.11`," "the retry ceiling is 5," "`X` still +lives in `Y`." A large fraction of these are **facts no test checks and CI stays green on**. They read +perfectly forever and rot silently. Dorian has the agent emit a tiny `claims.json` for the *checkable* +subset, seals only the currently-true ones into a git `.warrant`, and **re-checks them when the +watched files drift** — flipping to `REVOKED` (exit 4) and naming the exact broken claim. Token-free, +local-first, deterministic. + +This is two wins from one act, and the *first* is the one the strongest evidence supports. **(1) At +authoring time**, the seal is **born-verifiable**: it refuses to seal any claim that is *already* false +right now (exit 4) — directly catching the "**phantom change**" failure where an agent's description +asserts an edit the code does not actually contain, which an academic study finds is the single most +common message-code inconsistency (**~45%**, [arXiv 2601.04886](https://arxiv.org/abs/2601.04886)). +**(2) Later**, the same warrant is the *alarm* when a once-true fact silently becomes false. The first +catches the lie made now; the second catches the fact that rots. + +### One sentence +> **Dorian turns the checkable claims in your AI coding agent's change summary into deterministic git +> receipts that revoke when a later commit silently breaks them — especially the config, signature, and +> constant facts your tests and CI never check.** + +## Primary user + +**The solo Claude Code / AI-agent power user** on a trusted repo (then: the OSS maintainer / team lead +accepting AI-assisted PRs). Evidence ranks this segment highest on immediate pain × lowest adoption +friction: "agents routinely claim work they did not do, and the user feels it directly" +([dev.to](https://dev.to/moonrunnerkc/ai-coding-agents-lie-about-their-work-outcome-based-verification-catches-it-12b4)), +while OSS maintainers have the most *acute and public* pain but higher friction (they must get +contributors to comply) ([GitHub community](https://github.com/orgs/community/discussions/185387)). + +**Exact workflow (before → after):** +- *Before:* the agent ends with a summary; the human skims it, trusts the green check, and merges. A + later, unrelated refactor changes a default or a config value the summary asserted; nothing fails; + the summary is now a quiet lie that misleads the next reader (and the next agent that ingests it). +- *After:* the agent emits `claims.json`; `dorian verify … --strength-gate=fail` seals the true subset + into `.warrant` (committed); on every later PR the `dorian` Action runs `revalidate --since + base`; a broken claim folds the warrant to `REVOKED` and **blocks the PR**, naming the claim. + +## The "hair-on-fire" moment + +A later commit changes behavior or config that an earlier agent summary (or `CLAUDE.md`, or a doc, or a +release note) asserted — **no test fails, CI is green, the summary still reads perfectly** — and the +team ships on a false statement that the *next* agent then treats as ground truth. This is the +documented "**coding against fiction**" failure: after ~20 commits half the documented API surface no +longer exists, so every later agent suggestion is "built on a foundation of lies" +([dev.to](https://dev.to/mossrussell/your-ai-agent-is-coding-against-fiction-how-i-fixed-doc-drift-with-a-pre-commit-hook-1acn)). + +## Why Dorian specifically + +The mechanism maps 1:1 to the wedge: +- **Checkable claims + born-verifiable sealing** — only a falsifiable assertion gets a warrant; the + seal refuses anything already false (exit 4). This shifts the burden to the claimant to be checkable + — a structural answer to the "slop" flood. +- **Deterministic, token-free checkers** — a re-check is a `git show` + an `ast`/`tomllib` parse, not + an LLM re-judging. Identical run-to-run; **costs no tokens** while LLM review moves to *metered* + billing ([GitHub changelog, Apr 2026 — effective June 1](https://github.blog/changelog/2026-04-27-github-copilot-code-review-will-start-consuming-github-actions-minutes-on-june-1-2026/)). +- **Git-native `.warrant` sidecars + revalidation on drift** — the claim is persisted next to the code + and **automatically re-checked when the watched (and statically-imported) files change**. This is the + empty cell in the market: nothing else persists a *specific natural-language claim* and re-checks + *that exact claim* on drift. +- **Two-axis honesty (trigger vs truth)** — `--binding-gate` audits *when* a claim re-checks; + `--strength-gate` audits *whether* its checker can falsify it. Weak ≠ false. This is the antidote to + the #1 reason teams abandon AI-review tools — **noise / "cry wolf"** + ([codeant.ai](https://www.codeant.ai/blogs/prevent-ai-code-review-overload)): Dorian fires only when + a *specific sealed claim* deterministically breaks. + +## Why alternatives are insufficient + +| Substitute | Why it does not cover this wedge | +|---|---| +| **tests / coverage / CI** | Verify the *code*, not the *summary's specific claim*. The whole pain is the facts with **no failing test** (config, signature defaults, constants) — green CI is exactly what hides them. | +| **LLM PR reviewers** (CodeRabbit, Greptile, Copilot review) | Non-deterministic, **spend tokens every run** (now metered), re-judge from scratch, and **do not persist a sealed claim that fails on a *future* unrelated PR's drift**. Noise drives abandonment. | +| **SAST** (Semgrep, CodeQL) | Deterministic, but find *generic* vulnerability/quality patterns — not "is *this stated fact* still true?" | +| **in-toto / SLSA / Sigstore** | Persist *signed* claims, but about **build lineage / artifact provenance**, not claim-level truth that re-checks on source drift. Complementary, not substitute. | +| **policy-as-code** (OPA/Conftest) | Strongest deterministic/local-first peer, but encodes *generic* policy gates, not a *specific NL claim* bound to a content digest and re-checked on drift. | +| **doc-linters / CODEOWNERS** | Prose style and human ownership, not factual truth of a claim. | + +## Evidence from deep research (highest-confidence) + +- **Agent PRs now dominate review and "look correct" is breaking:** >1 in 5 GitHub code reviews involve + an agent; GitHub names "**Hallucinated Correctness**" (passes tests, wrong logic) and "**CI Gaming**" + (removed tests / weakened thresholds) as top red flags + ([GitHub Blog, June 2026](https://github.blog/ai-and-ml/generative-ai/agent-pull-requests-are-everywhere-heres-how-to-review-them/)). +- **"Phantom Changes":** an academic study finds AI PR descriptions routinely **claim changes the code + does not contain** ([arXiv 2601.04886](https://arxiv.org/abs/2601.04886)). GitHub's own docs admit + Copilot PR summaries "can be inaccurate" and are **not auto-refreshed when the PR changes** — they go + stale ([GitHub docs](https://docs.github.com/en/copilot/responsible-use/pull-request-summaries)). +- **Verification debt is quantified:** incidents/PR **+23.5%**, change-failure rate **~+30%**, only + **45%** of teams have an AI policy ([Cortex 2026 Benchmark](https://www.cortex.io/post/ai-is-making-engineering-faster-but-not-better-state-of-ai-benchmark-2026)). +- **The triage cost doesn't scale:** curl's maintainer reports an AI-"slop" flood — ~20% of bug-bounty + submissions AI-generated, only ~5% genuine, each still engaging 3–4 humans — i.e. human triage that + does not scale with AI volume + ([Daniel Stenberg, July 2025](https://daniel.haxx.se/blog/2025/07/14/death-by-a-thousand-slops/)). +- **The market wants deterministic, fast, stateless gates,** not more probabilistic noise + ([Sourcegraph](https://sourcegraph.com/blog/automated-code-review-tools); + [Stack Overflow 2025](https://survey.stackoverflow.co/2025/ai)). + +(Full ledger: 37 sources in `.claude/usecase_sources.jsonl`; not all are committed.) + +## Evidence from real-project trials (first-hand, this session) + +All on the **published `dorian-vwp` 1.2.0** (clean `pip install`, zero deps). + +1. **Killer trial — agent change-note on a real repo.** `python-dotenv` @ `751f8c148222…`. A realistic + agent summary made 4 claims (3 checkable + 1 prose). Dorian sealed the 3 checkable ones under + `--strength-gate=fail` (`py-signature` of `load_dotenv` incl. `override=False`; `config-value` + `requires-python ">=3.10"`; `symbol` `dotenv_values`); `--binding-gate=warn` honestly noted + `load_dotenv` is also mentioned in CHANGELOG/README. A later commit flipped `override=False → True` + → `revalidate` → **BROKEN** (`signature_mismatch: default of 'override': 'True' != 'False'`) → + **WARRANTED → REVOKED, exit 4**. The summary file never changed; the repo's own tests/CI stay silent. + `py-signature` caught a **default-value** drift a `symbol:` existence check would miss. +2. **Behavior receipt (C4).** Scratch repo; a `pytest:`-backed behavior claim sealed under + `--strength-gate=fail`; broke the implementation → `revalidate` → **BROKEN → REVOKED, exit 4**. +3. **Honest limit (C4 env).** Backing a C4 claim on `python-dotenv`'s own test failed collection + (missing `click`) → `ERRORED_AT_SEAL` (fail-closed, never a false pass). Structural C3 claims need + no deps and are the low-friction default. + +Detail: `docs/OUTSIDE_WORLD_VALIDATION.md` and `.claude/usecase_trials.jsonl`. **Every drift is a +deliberate validation mutation, not an organic catch.** + +## Demo script + +[`DEMO_SCRIPT_BEST_USE_CASE.md`](DEMO_SCRIPT_BEST_USE_CASE.md) — copy-paste, <3 min, clean install → +seal → drift → `REVOKED`, no LLM calls at check time. (It is trial #1 above, generalized.) + +## Adoption path (the wedge distribution channel) + +**A Claude Code final-message convention + the `dorian` GitHub Action.** The agent ends a turn with a +small `claims.json` (template in [`CLAUDE_CODE_DORIAN_WORKFLOW.md`](CLAUDE_CODE_DORIAN_WORKFLOW.md)); +`dorian verify` seals; the Action (`fail_on: revoked`) re-checks on every later PR and posts a +deterministic comment. Lowest-friction beachhead: the solo power user who already runs Claude Code. +Second hop: a template repo + the "agent must emit `claims.json`" convention for OSS/teams. + +## What to build next (grounded — no SaaS/dashboard) + +1. **A Claude Code skill/hook that drafts `claims.json` from the agent's final message** (deterministic + scaffolding, human-reviewed) — collapses the authoring cost the critics flagged. +2. **Warrant signing (optional).** Today warrants are content-addressed but **unsigned** — a sidecar is + forgeable by anyone with repo-write access. An optional `--sign` (e.g. Sigstore/cosign) closes the + one real "receipt you can trust" gap and is the honest complement to in-toto/SLSA. +3. **Base-ref selection for `--checker-source base`** (already tracked) — hardens the fork story. +4. **One or two more structural checkers where evidence is strong** (e.g. YAML config-value) — only if + demand shows up; keep the grammar small. + +## What NOT to build (tempting but wrong) + +- An **LLM judge / semantic PR reviewer** — that is the crowded, non-deterministic, token-spending lane + Dorian explicitly wins by avoiding. +- A **SaaS dashboard** — the local-first, git-native sidecar *is* the product. +- A **sandbox** — C4/C5 execute code; stay trusted-repo-scoped and say so. +- **Auto-extracting claims from arbitrary prose with a model at seal time** — keep extraction + draft-only; the agent (or human) authors the claims. +- Marketing the wedge as covering the *whole* agent summary — it covers the **checkable** subset, and + is *differentiated* most on the **no-test facts**. Overclaiming behavioral coverage is the fastest way + to lose a skeptical developer. + +## Risks and honest limits + +- **Truth ≤ checker strength** (strongest is `pytest`); cheap checkers (`symbol:`/`string:`) prove + existence, not behavior. Surfaced by `--strength-gate`, not hidden. +- **Only verifies claims someone *wrote*** — no lies of omission; Dorian produces no claims or code. +- **Unsigned warrants** — forgeable with repo-write access (see "build next"). +- **Python/format-centric structural depth** (`py-signature`/`py-const` are Python-AST; `config-value` + is TOML/JSON). +- **Authoring discipline** — `claims.json` is a new habit; the win must exceed the cost (hence the + drafting-hook recommendation). +- **Trusted-repo scope; not a sandbox.** + +## Final recommendation + +Ship the wedge as **"the receipt for what your coding agent claimed it changed — and the alarm when a +later commit makes it a lie,"** aimed first at the **solo Claude Code power user**, distributed through +a **final-message `claims.json` convention + the GitHub Action**, and **honest that its differentiated +core is the no-test facts** (config / signature / constant / reference drift) that tests, SAST, +coverage, and CI structurally cannot catch. Secondary, framed as *the same mechanism*: PR-summary rot +(B) and executable-docs drift (E). Do not chase the LLM-judge, SaaS, sandbox, or compliance lanes. diff --git a/docs/CLAUDE_CODE_DORIAN_WORKFLOW.md b/docs/CLAUDE_CODE_DORIAN_WORKFLOW.md new file mode 100644 index 0000000..8b48004 --- /dev/null +++ b/docs/CLAUDE_CODE_DORIAN_WORKFLOW.md @@ -0,0 +1,99 @@ +# Claude Code + Dorian — turn your agent's summary into receipts + +> A practical workflow for Claude Code / Codex / Cursor users. When the agent finishes a change, it +> ends with a summary of specific claims. This is how to make the **checkable** subset into deterministic +> git receipts that fail later if the code drifts — token-free, in a trusted repo. + +## The loop + +1. The agent makes the change and writes its normal summary / PR description. +2. The agent emits a small `claims.json` for **only the checkable** statements in that summary. +3. `dorian verify --claims claims.json --strength-gate=fail` seals a `.warrant` sidecar + (born-verifiable: it refuses to seal anything that is already false). +4. Commit the `.warrant` alongside the change. +5. On every later PR, the GitHub Action runs `dorian revalidate --since `; if a later edit broke a + sealed claim, the warrant folds to **REVOKED** (exit 4) and the Action blocks — naming the claim. + +## How the agent should write claims + +Pick the **checker whose strength matches the claim** (the truth axis). Each claim is +`{id, text, kind, load_bearing, checkers:[{type, program}]}`. + +| The summary says… | kind | checker (`program`) | +|---|---|---| +| "function `f` exists in `m.py`" | reference | `C3` `symbol:m.py::f` | +| "`f`'s signature/defaults are `(a, b=1) -> int`" | reference | `C3` `py-signature:m.py::f::a, b=1 -> int` | +| "the constant `TIMEOUT` is `30`" | quantity | `C3` `py-const:m.py::TIMEOUT::30` | +| "`requires-python` is `>=3.11`" / a config default | quantity | `C3` `config-value:pyproject.toml:project.requires-python:">=3.11"` | +| "the string/route `"/admin"` is present in `r.py`" | reference | `C3` `string:r.py::/admin` or `regex:` / `code:` | +| "behavior B holds (covered by test T)" | behavior | `C4` `pytest:tests/test_x.py::T` | +| "the data file has ≥N rows / column C / freshness" | quantity/fact | `C5` `rowcount:`/`schema:`/`freshness:` | + +Rule of thumb: a **behavior** claim needs a `C4` test (or it is under-verified); a **quantity** claim +needs a value-pinning checker (`py-const`/`config-value`/anchored `regex`/typed `C5`), not a bare +existence check. `--strength-gate=fail` enforces exactly this for load-bearing claims. + +## Good vs bad claims + +**Good (checkable, specific, falsifiable):** +- `py-signature:src/auth.py::verify_token::token, algo="RS256" -> bool` +- `config-value:pyproject.toml:project.requires-python:">=3.11"` +- `pytest:tests/test_auth.py::test_rs256_roundtrip` for "RS256 tokens verify" + +**Bad (do not seal these — they are prose, not receipts):** +- "the code is cleaner / more maintainable / more performant" (not checkable) +- "this is a big improvement" (opinion) +- a **behavior** claim backed only by `symbol:` (exists ≠ behaves — `--strength-gate=fail` refuses it) +- vague "updated the docs" with no specific anchor + +## Running Dorian + +```bash +# one-shot: auto-capture the read-set from the claims, run checkers, seal +dorian verify dorian-change-note.md --claims claims.json --strength-gate=fail --binding-gate=warn + +# later, on a PR: +dorian revalidate --since origin/main # REVOKED (exit 4) if a sealed claim broke +dorian status # trust state of every warranted artifact +``` + +`dorian init` scaffolds a starter `claims.json`, a change note, and a `.github/workflows/dorian.yml`. + +## GitHub Action + +```yaml +- uses: ajaysurya1221/dorian/action@v1.2.0 + with: + fail_on: revoked # block the PR when a sealed claim breaks + # for semi-trusted contributors, also: checker_source: base + deny_exec: true +``` + +It posts a customer-readable PR comment (`Blocked/Passed/Errored`, the claim that changed, the affected +`.warrant`). The check is deterministic and token-free. + +## What to avoid + +- Don't seal marketing/opinion claims — only checkable facts. +- Don't use Dorian as a sandbox: `C4 pytest:` and `C5 shell:` execute code; use it in **trusted** repos, + and for semi-trusted contributors combine `checker_source: base` + `deny_exec: true` (see + [`SECURITY_BOUNDARY.md`](SECURITY_BOUNDARY.md)). +- Don't read a weak-binding or weak-strength warning as "the claim is false" — it means low coverage/ + confidence; strengthen the watch or the checker. + +## Recommended Claude Code final-message template + +End a coding turn with both a human summary and a machine block: + +``` +### What I changed + + +### Checkable receipts (dorian) +I emit claims.json with these load-bearing, checker-backed claims: +- : +(only statements a deterministic checker can falsify; prose is excluded) +Run: dorian verify --claims claims.json --strength-gate=fail +``` + +This keeps the agent honest at the moment of maximum context, and leaves a receipt that fails later if +the code drifts — without spending a single token to re-check. diff --git a/docs/DEMO_SCRIPT_BEST_USE_CASE.md b/docs/DEMO_SCRIPT_BEST_USE_CASE.md new file mode 100644 index 0000000..5cd03bd --- /dev/null +++ b/docs/DEMO_SCRIPT_BEST_USE_CASE.md @@ -0,0 +1,102 @@ +# Demo — "Receipts for what your AI coding agent claimed it changed" + +> A copy-paste demo, **under 3 minutes**, from a clean `pip install`. It seals the *checkable* claims +> from an AI agent's change summary into a git `.warrant`, then shows a later, unrelated change flip the +> warrant to **REVOKED** — naming the exact stale claim. **Zero LLM calls at check time.** Every command +> below was run first-hand on 2026-06-27 against `python-dotenv` at the pinned SHA. + +## 0. Install (clean, zero runtime deps) + +```bash +pip install dorian-vwp # -> dorian 1.2.0 ; core has zero runtime dependencies +dorian --version +``` + +## 1. A real repo + a realistic agent change-note + +```bash +git clone --depth 1 https://github.com/theskumar/python-dotenv.git && cd python-dotenv +git rev-parse HEAD # 751f8c148222e58aa173c83c4e5e6cfccb2cc124 at time of writing + +cat > AGENT_CHANGE_NOTE.md <<'MD' +## Summary (from the coding agent) +- `load_dotenv()` keeps its public signature, including `override=False` by default. [checkable] +- The package still targets Python `>=3.10`. [checkable] +- `dotenv_values()` remains the canonical parser entry point. [checkable] +- I made the code "cleaner and more maintainable". [NOT checkable — prose, left out] +MD +``` + +## 2. The agent emits receipts — a `claims.json` for the *checkable* subset only + +```bash +cat > claims.json <<'JSON' +{"claims": [ + {"id": "load-dotenv-default-override-false", + "text": "load_dotenv keeps override=False by default in its public signature.", + "kind": "reference", "load_bearing": true, + "checkers": [{"type": "C3", "program": "py-signature:src/dotenv/main.py::load_dotenv::dotenv_path=None, stream=None, verbose=False, override=False, interpolate=True, encoding=\"utf-8\""}]}, + {"id": "requires-python-3-10", + "text": "The package targets Python >=3.10.", + "kind": "quantity", "load_bearing": true, + "checkers": [{"type": "C3", "program": "config-value:pyproject.toml:project.requires-python:\">=3.10\""}]}, + {"id": "dotenv-values-exists", + "text": "dotenv_values() is the canonical parser entry point.", + "kind": "reference", "load_bearing": true, + "checkers": [{"type": "C3", "program": "symbol:src/dotenv/main.py::dotenv_values"}]} +]} +JSON +``` + +## 3. Seal — born-verifiable (refuses to seal anything false right now) + +```bash +dorian verify AGENT_CHANGE_NOTE.md --claims claims.json --strength-gate=fail --binding-gate=warn +# verified 3/3 claim(s) -> AGENT_CHANGE_NOTE.md.warrant (exit 0, WARRANTED) +``` + +What you see: `--strength-gate=fail` confirms each load-bearing claim's checker can actually falsify it +(0 high-risk). `--binding-gate=warn` honestly notes `load_dotenv` is *also* mentioned in CHANGELOG/README +— a trigger-coverage hint, **not** a claim being false. The `.warrant` is a deterministic git sidecar. + +## 4. Drift — a later change flips the default the summary claimed + +```bash +# simulate a future, unrelated edit that changes behavior the summary asserted +sed -i.bak 's/ override: bool = False,/ override: bool = True,/' src/dotenv/main.py +git add -A && git commit -m "change override default" +``` + +`AGENT_CHANGE_NOTE.md` never changed and still reads perfectly. The repo's own tests/CI don't know the +*summary* made a promise about `override`. + +## 5. Revalidate — the stale receipt REVOKES, by name + +```bash +dorian revalidate --since HEAD~1 +# BROKEN load-dotenv-default-override-false C3: signature_mismatch: load_dotenv: default of 'override': 'True' != expected 'False' +# VERIFIED dotenv-values-exists +# fold WARRANTED -> REVOKED (exit 4) +``` + +`exit 4` is the gate. The default `dorian` GitHub Action (`fail_on: revoked`) turns this into a blocked +PR — so a summary claim that silently stopped being true cannot ship unnoticed. **No tokens were spent +to re-check** — it is a `git show` + an `ast` parse. + +## The one-screen point + +| | Without Dorian | With Dorian | +|---|---|---| +| The agent's summary | prose nobody re-checks | the checkable subset becomes git receipts | +| A later refactor breaks a claim | summary silently rots; tests stay green | warrant flips **REVOKED**, names the claim | +| Re-check cost | a human re-reads, or an LLM re-judges (tokens) | deterministic, token-free, milliseconds | + +## Notes / honest limits + +- `py-signature` caught a **default-value** change that a `symbol:` existence check would miss — pick the + checker strength that matches the claim (that is the *truth axis*; `--strength-gate` audits it). +- A **C4 `pytest:`** behavior claim is even stronger, but it needs the target's test env to be runnable + (missing test deps → `ERRORED_AT_SEAL`, fail-closed — never a false pass). Structural C3 claims + (`py-signature`/`config-value`/`symbol`) need no deps and are the low-friction default. +- Dorian verifies **specific, checkable** claims in **trusted** repos. It is not a sandbox, not an LLM + judge, and not a replacement for tests/review. See [`SECURITY_BOUNDARY.md`](SECURITY_BOUNDARY.md). diff --git a/docs/POSITIONING_2026_06_27.md b/docs/POSITIONING_2026_06_27.md new file mode 100644 index 0000000..4f20180 --- /dev/null +++ b/docs/POSITIONING_2026_06_27.md @@ -0,0 +1,98 @@ +# Positioning — 27 June 2026 + +> Sharp, honest positioning for Dorian's best use case (see +> [`BEST_USE_CASE_2026_06_27.md`](BEST_USE_CASE_2026_06_27.md)). Every line is constrained by +> [`VALIDATION_HONESTY.md`](VALIDATION_HONESTY.md): no sandbox, no LLM judging, no semantic proof, no +> broad compliance, not a replacement for tests/review. + +## Primary tagline + +> **Receipts for what your AI coding agent claimed it changed.** + +Sub-line (when there's room): + +> The checkable claims in your agent's change summary become deterministic git receipts that **revoke +> when a later commit makes one false** — token-free, in your own repo. + +## Landing-page hero + +> **Your coding agent's summary reads perfectly. Six commits later, half of it is quietly false.** +> +> Dorian turns the *checkable* claims an AI coding agent makes — a signature default, a config value, a +> constant, "X still lives in Y" — into deterministic receipts sealed into git. When a later change +> silently breaks one, Dorian flips it to **REVOKED** and names it. No tokens at check time. No LLM +> judging. Your repo, your machine. +> +> `pip install dorian-vwp` · CLI + GitHub Action · zero runtime dependencies + +## README pitch (option) + +> **Dorian — hold your AI coding agent to what it said it changed.** +> An agent finishes a change and summarizes it. Most of that summary is facts **no test checks** — +> signatures, config defaults, constants, references — so they rot silently as the code drifts. Dorian +> seals the checkable subset into a git `.warrant` (born-verifiable: it won't seal anything already +> false), then **re-checks exactly those claims when the watched files change** — deterministically, +> token-free. A broken claim folds the warrant to `REVOKED` (exit 4) and blocks the PR, naming the +> claim. Not an LLM judge, not a sandbox, not a replacement for tests — the receipt layer that keeps +> stated facts honest as the code moves underneath them. + +## GitHub release blurb + +> Dorian is the receipt for what your AI coding agent claimed it changed. Seal the checkable claims from +> a change summary into git `.warrant` sidecars; Dorian re-checks them on drift and `REVOKE`s the ones a +> later commit broke — deterministic, local-first, **token-free**. Best for the no-test facts (config, +> signatures, constants, references) your CI stays green on. `pip install dorian-vwp`. + +## 5 post drafts + +1. Your AI agent's PR summary isn't auto-refreshed when the PR changes — GitHub says so in its own + docs. So it goes stale, and the next agent treats the stale summary as ground truth. Dorian seals the + *checkable* claims into git and revokes them when the code drifts. Token-free. `pip install dorian-vwp` + +2. Tests verify your code. Nothing verifies your agent's *claim* that "the default is still False" — no + test fails when it flips. That's the gap Dorian fills: a deterministic receipt that REVOKES on drift. + Real demo (python-dotenv, 3 min) → [demo link]. + +3. LLM PR reviewers re-judge from scratch every run and now bill you per review in tokens + CI minutes. + Dorian re-checks a *specific sealed claim* with a `git show` + an `ast` parse — same answer every + time, zero tokens. Deterministic > probabilistic for a hard gate. + +4. "Phantom changes" — AI PR descriptions that claim edits the code doesn't contain — are now a measured + phenomenon (arXiv 2601.04886). Make your agent emit `claims.json`; Dorian refuses to seal anything + that isn't actually true *right now*, and revokes it later if it stops being true. + +5. Dorian isn't a sandbox, an LLM judge, or a dashboard. It's a 1-file-per-claim git sidecar that says + "this stated fact was true at commit X" and yells when a later commit makes it false. Boring, + deterministic, token-free. That's the point. + +## Maintainer-facing explanation + +You're merging AI-assisted PRs whose summaries you can't fully re-derive by hand. Ask the change to ship +a `claims.json` of its load-bearing, *checkable* facts (signatures, config, constants, references, and +behavior backed by a `pytest:` node). `dorian verify --strength-gate=fail` seals only the true ones; the +committed `.warrant` then re-checks on every later PR via the Action. You get a deterministic, +low-noise signal — `REVOKED` names the exact claim — instead of re-reading a 30-line summary or trusting +a green check. It complements your tests and review; it does not replace them, and it is scoped to your +own trusted repo (not untrusted forks). + +## Enterprise / platform-engineer explanation + +Standardize "every risky agent-authored change emits checkable claims" across your trusted internal +repos. Dorian is a local-first, **token-free**, deterministic gate that persists claim-level truth in +git and re-checks on drift — a primitive that *scales with PR volume* without per-review model cost +(unlike LLM reviewers, which are moving to metered billing). It **complements** SLSA/in-toto build +provenance (lineage) with claim-level truth, and policy-as-code (generic gates) with *specific stated +facts*. Roll it out bottom-up from power users; it is not a SaaS and stores nothing off your machine. + +## Anti-hype disclaimers (always attach) + +- Dorian verifies **specific, checkable** claims — **not** arbitrary correctness, design, or taste, and + it has **no semantic understanding** of prose. +- It is **not a sandbox** (`C4 pytest:`/`C5 shell:` execute code; trusted repos only), **not an LLM + judge**, **not a replacement** for tests/SAST/code review/human judgment. +- Truth is only as strong as the checker behind a claim (strongest is `pytest`); weak binding or weak + checker strength means **low confidence, not a false claim**. +- It verifies only claims someone **wrote** — it cannot catch a lie of omission. +- Warrants are content-addressed but **unsigned** today — trust the repo's write access accordingly. +- Evidence so far is one documented real cross-PR catch plus synthetic/scoped benchmarks and the trials + in `OUTSIDE_WORLD_VALIDATION.md` — **not** broad market validation. diff --git a/docs/USE_CASE_DECISION_MATRIX_2026_06_27.md b/docs/USE_CASE_DECISION_MATRIX_2026_06_27.md new file mode 100644 index 0000000..3eca1d0 --- /dev/null +++ b/docs/USE_CASE_DECISION_MATRIX_2026_06_27.md @@ -0,0 +1,76 @@ +# Use-Case Decision Matrix — 27 June 2026 + +> All eight candidate wedges, scored against the rubric by independent strategist agents (fed the +> 37-source research digest + competitive matrix), then adversarially critiqued. Companion to +> [`BEST_USE_CASE_2026_06_27.md`](BEST_USE_CASE_2026_06_27.md). + +## Rubric & weights + +Each dimension scored 1–5. For `overclaiming_risk` and `implementation_gap`, higher = worse. + +``` +weighted = 2.0*pain + 1.5*dorian_fit + 1.5*demo_clarity + 1.2*adoption_friction_inverse + + 1.2*evidence_strength + 1.0*differentiation + 1.0*frequency + 0.8*urgency + + 0.8*trust_boundary_fit + 0.5*expansion_potential + - 1.5*overclaiming_risk - 1.0*implementation_gap +``` + +## Scores (ranked) + +| Wedge | W | pain | fit | demo | friction⁻¹ | evid | diff | freq | urg | trust | exp | over‑claim | gap | verdict | +|---|--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|---| +| **A. AI-agent change-note verifier** | **63.7** | 5 | 5 | 5 | 4 | 5 | 5 | 5 | 4 | 4 | 5 | 3 | 1 | **PRIMARY** | +| E. Docs-drift detector | 59.7 | 4 | 5 | 5 | 3 | 4 | 4 | 4 | 3 | 5 | 4 | 2 | 2 | → fold into A (supporting) | +| B. PR-summary rot detector | 52.7 | 5 | 5 | 5 | 3 | 5 | 5 | 4 | 4 | 4 | 4 | 3 | 2 | **SECONDARY** | +| D. Platform-team AI governance | 42.7 | 5 | 5 | 3 | 2 | 4 | 5 | 4 | 4 | 5 | 5 | 3 | 4 | expansion (later) | +| C. Release-note truth gate | 42.5 | 3 | 5 | 5 | 4 | 3 | 4 | 2 | 3 | 5 | 4 | 2 | 1 | niche-but-clean | +| F. Compliance/provenance sidecar | 28.4 | 2 | 3 | 2 | 2 | 2 | 4 | 2 | 2 | 4 | 3 | 4 | 4 | **REJECT** | +| G. ML/data benchmark verifier | 23.3 | 3 | 3 | 3 | 2 | 1 | 3 | 2 | 2 | 2 | 3 | 4 | 4 | **REJECT** | +| H. OSS untrusted-contribution tool | 22.1 | 4 | 2 | 3 | 1 | 4 | 3 | 3 | 3 | 1 | 2 | 5 | 4 | **REJECT** | + +## Synthesis + +**Primary: A — refined.** Highest on pain, fit, demo, evidence, differentiation, frequency, and lowest +implementation gap (1). The adversarial critique kept it but **narrowed the framing**: a real agent +summary is *dominated* by behavioral/"tests cover X" claims, so do **not** pitch "verify the whole +summary." The defensible, differentiated core is the **no-failing-test facts** (config, packaging, +signatures, constants, file/symbol references) that **CI stays green on** when they drift — exactly the +gap tests/SAST/coverage cannot close. Adopt A with that honest scope. + +**Secondary: B (PR-summary rot)** — same mechanism, team/maintainer framing; critique: re-scope from +"summary rot" to "load-bearing config/packaging/signature facts an agent asserts," because Dorian +re-checks only the *checkable* subset, not the prose. + +**Fold in: E (docs-drift)** — scored #2 but the critique **downgraded it to a supporting case** because +"docs-drift detector" overclaims the mechanism: Dorian does **not** read docs and discover assertions; +a human/agent writes the claims. It is the *same* receipt mechanism pointed at a doc. Keep it as a +sub-use of A, not a separate headline. + +**Expansion (not now): D (platform governance)** — strong pain/fit but demo=3, friction⁻¹=2, gap=4: a +top-down rollout, not a wedge. It is where A *expands* after bottom-up adoption. + +**Clean niche: C (release-note truth gate)** — low frequency (releases are rare) but very clean demo +and trust fit; a good secondary marketing artifact, not the beachhead. + +## Rejections (with reasons) + +- **F. Compliance/provenance sidecar (28.4)** — wrong buyer (enterprise platform/security teams running + attestation pipelines, not the local power user); trust-boundary mismatch; high overclaim/gap. + in-toto/SLSA already own *build* provenance; Dorian's `export --in-toto` is experimental. **Complement, + don't position here.** +- **G. ML/data benchmark verifier (23.3)** — branding it a "metric verifier" overclaims model-quality + adjudication Dorian cannot do; metric claims reduce to a `confidence`-free threshold check. Weakest + evidence (1). The C5 *schema/rowcount/freshness* capability is real but a feature, not the wedge. +- **H. OSS untrusted-contribution tool (22.1)** — **trust-boundary inversion**: the value prop asks the + maintainer to trust a warrant produced by an *untrusted* contributor, violating Dorian's trusted-repo + model (and `.warrant` files are unsigned/forgeable). Highest overclaim risk (5). Dorian helps the + maintainer's **own** agent-assisted work, not untrusted fork PRs. + +## The one-line test + +A wins because it is the only wedge where all of these are simultaneously true: the pain is **named and +growing** (agent summaries go stale; "phantom changes"), Dorian's mechanism is an **exact fit** +(persist a specific claim → re-check on drift, token-free), the **demo lands in 3 minutes**, adoption is +**one `claims.json` + a GitHub Action**, and the differentiation answer ("why not tests/SAST/LLM-bot?") +is **clean** — *nothing else re-checks a specific stated fact after later drift, deterministically and +for free.* diff --git a/docs/changes/best-use-case-discovery.claims.json b/docs/changes/best-use-case-discovery.claims.json new file mode 100644 index 0000000..80ea490 --- /dev/null +++ b/docs/changes/best-use-case-discovery.claims.json @@ -0,0 +1,46 @@ +{ + "claims": [ + { + "id": "pyproject-version-1-2-0", + "text": "dorian-vwp is version 1.2.0 (the version the demo installs).", + "kind": "quantity", + "load_bearing": true, + "checkers": [{ "type": "C3", "program": "config-value:pyproject.toml:project.version:\"1.2.0\"" }] + }, + { + "id": "zero-runtime-dependencies", + "text": "dorian's core ships zero runtime dependencies.", + "kind": "quantity", + "load_bearing": true, + "checkers": [{ "type": "C3", "program": "config-value:pyproject.toml:project.dependencies:[]" }] + }, + { + "id": "cli-exposes-revalidate", + "text": "The CLI exposes the `revalidate` command the demo's drift step uses.", + "kind": "reference", + "load_bearing": true, + "checkers": [{ "type": "C3", "program": "regex:src/dorian/cli.py::add_parser\\(\"revalidate\"" }] + }, + { + "id": "security-boundary-not-a-sandbox", + "text": "SECURITY_BOUNDARY.md states the not-a-sandbox boundary the positioning relies on.", + "kind": "reference", + "load_bearing": true, + "checkers": [{ "type": "C3", "program": "regex:docs/SECURITY_BOUNDARY.md::Not a sandbox" }] + }, + { + "id": "best-use-case-doc-exists", + "text": "The best-use-case deliverable exists.", + "kind": "reference", + "load_bearing": true, + "checkers": [{ "type": "C3", "program": "path:docs/BEST_USE_CASE_2026_06_27.md" }] + }, + { + "id": "demo-script-doc-exists", + "text": "The demo script deliverable exists.", + "kind": "reference", + "load_bearing": true, + "checkers": [{ "type": "C3", "program": "path:docs/DEMO_SCRIPT_BEST_USE_CASE.md" }] + } + ] +} diff --git a/docs/changes/best-use-case-discovery.md b/docs/changes/best-use-case-discovery.md new file mode 100644 index 0000000..85f5116 --- /dev/null +++ b/docs/changes/best-use-case-discovery.md @@ -0,0 +1,16 @@ +# Change note — best-use-case discovery (2026-06-27) + +Adds the evidence-backed use-case package: +[`BEST_USE_CASE_2026_06_27.md`](../BEST_USE_CASE_2026_06_27.md), +[`USE_CASE_DECISION_MATRIX_2026_06_27.md`](../USE_CASE_DECISION_MATRIX_2026_06_27.md), +[`DEMO_SCRIPT_BEST_USE_CASE.md`](../DEMO_SCRIPT_BEST_USE_CASE.md), +[`CLAUDE_CODE_DORIAN_WORKFLOW.md`](../CLAUDE_CODE_DORIAN_WORKFLOW.md), and +[`POSITIONING_2026_06_27.md`](../POSITIONING_2026_06_27.md). The chosen wedge: **receipts for the +checkable claims an AI coding agent makes — especially the no-failing-test facts (config, signatures, +constants, references) — that REVOKE on drift**. No code, grammar, or security-posture change. + +Dogfooded: the load-bearing facts the docs and demo rely on are sealed in +[`best-use-case-discovery.claims.json`](best-use-case-discovery.claims.json) under +`--strength-gate=fail` — the package is v1.2.0 with a zero-dependency core, the CLI exposes the +`revalidate` command the demo uses, and `SECURITY_BOUNDARY.md` states the not-a-sandbox boundary the +positioning leans on. diff --git a/docs/changes/best-use-case-discovery.md.warrant b/docs/changes/best-use-case-discovery.md.warrant new file mode 100644 index 0000000..c1d8a3e --- /dev/null +++ b/docs/changes/best-use-case-discovery.md.warrant @@ -0,0 +1,192 @@ +{ + "spec_version": "0.1", + "warrant": { + "id": "sha256:17b3187f107c8d00f12c18062d0336c78cb6f0aa7c3c684af5fecc474bcd9766", + "artifact": { + "uri": "docs/changes/best-use-case-discovery.md", + "content_hash": "sha256:70ae592528abff9f2a7a52262aea4884c2666d4f9319cce366119c1c5fa13467", + "git_ref": "0e99d46ad84acb916fa3dc5b5655602bc878db7f" + }, + "produced_by": { + "runner": "manual", + "captured_at": "2026-06-27T14:47:21.681741+00:00", + "run_ref": "", + "model": "", + "prompt_hash": "" + }, + "read_set": [ + { + "id": "rs-0", + "uri": "pyproject.toml", + "selector": null, + "hash": "sha256:6a0c7b8ca3cebffe92270fb7f302f4f77ba722db696d435357eacd524249286d", + "version": "0e99d46ad84acb916fa3dc5b5655602bc878db7f", + "scope": "project" + }, + { + "id": "rs-1", + "uri": "src/dorian/cli.py", + "selector": null, + "hash": "sha256:f02d719cf5c318f9b92efc1727e8ed011a54ce7027736bcc701504592179279c", + "version": "0e99d46ad84acb916fa3dc5b5655602bc878db7f", + "scope": "project" + }, + { + "id": "rs-2", + "uri": "docs/SECURITY_BOUNDARY.md", + "selector": null, + "hash": "sha256:b606bcbf87d45f8e4abe3e533946e06890a36e6688270e53c09a7984cc0add74", + "version": "0e99d46ad84acb916fa3dc5b5655602bc878db7f", + "scope": "project" + }, + { + "id": "rs-3", + "uri": "docs/BEST_USE_CASE_2026_06_27.md", + "selector": null, + "hash": "sha256:50b287f6de60804c25df1e15b32aab215d122c1b916f625efd1056085b5c52c6", + "version": "0e99d46ad84acb916fa3dc5b5655602bc878db7f", + "scope": "project" + }, + { + "id": "rs-4", + "uri": "docs/DEMO_SCRIPT_BEST_USE_CASE.md", + "selector": null, + "hash": "sha256:02bd774fb2468b00a79974b453667d728a66f1fb1cb6cdaa722229d05fcffa60", + "version": "0e99d46ad84acb916fa3dc5b5655602bc878db7f", + "scope": "project" + }, + { + "id": "rs-5", + "uri": "src/dorian/revalidate.py", + "selector": null, + "hash": "sha256:3983ca4569cb8eca1f5f718d1a4476e19998c8f192586501d7390b5a18ef627b", + "version": "0e99d46ad84acb916fa3dc5b5655602bc878db7f", + "scope": "project" + } + ], + "claims": [ + { + "id": "pyproject-version-1-2-0", + "text": "dorian-vwp is version 1.2.0 (the version the demo installs).", + "kind": "quantity", + "anchor": null, + "load_bearing": true, + "supports": [], + "checkers": [ + { + "type": "C3", + "program": "config-value:pyproject.toml:project.version:\"1.2.0\"", + "expect": "exit:0", + "watch": [ + "pyproject.toml" + ], + "timeout_s": 30 + } + ] + }, + { + "id": "zero-runtime-dependencies", + "text": "dorian's core ships zero runtime dependencies.", + "kind": "quantity", + "anchor": null, + "load_bearing": true, + "supports": [], + "checkers": [ + { + "type": "C3", + "program": "config-value:pyproject.toml:project.dependencies:[]", + "expect": "exit:0", + "watch": [ + "pyproject.toml" + ], + "timeout_s": 30 + } + ] + }, + { + "id": "cli-exposes-revalidate", + "text": "The CLI exposes the `revalidate` command the demo's drift step uses.", + "kind": "reference", + "anchor": null, + "load_bearing": true, + "supports": [], + "checkers": [ + { + "type": "C3", + "program": "regex:src/dorian/cli.py::add_parser\\(\"revalidate\"", + "expect": "exit:0", + "watch": [ + "src/dorian/cli.py", + "src/dorian/revalidate.py" + ], + "timeout_s": 30 + } + ] + }, + { + "id": "security-boundary-not-a-sandbox", + "text": "SECURITY_BOUNDARY.md states the not-a-sandbox boundary the positioning relies on.", + "kind": "reference", + "anchor": null, + "load_bearing": true, + "supports": [], + "checkers": [ + { + "type": "C3", + "program": "regex:docs/SECURITY_BOUNDARY.md::Not a sandbox", + "expect": "exit:0", + "watch": [ + "docs/SECURITY_BOUNDARY.md" + ], + "timeout_s": 30 + } + ] + }, + { + "id": "best-use-case-doc-exists", + "text": "The best-use-case deliverable exists.", + "kind": "reference", + "anchor": null, + "load_bearing": true, + "supports": [], + "checkers": [ + { + "type": "C3", + "program": "path:docs/BEST_USE_CASE_2026_06_27.md", + "expect": "exit:0", + "watch": [ + "docs/BEST_USE_CASE_2026_06_27.md" + ], + "timeout_s": 30 + } + ] + }, + { + "id": "demo-script-doc-exists", + "text": "The demo script deliverable exists.", + "kind": "reference", + "anchor": null, + "load_bearing": true, + "supports": [], + "checkers": [ + { + "type": "C3", + "program": "path:docs/DEMO_SCRIPT_BEST_USE_CASE.md", + "expect": "exit:0", + "watch": [ + "docs/DEMO_SCRIPT_BEST_USE_CASE.md" + ], + "timeout_s": 30 + } + ] + } + ], + "derives_from": [], + "fold_policy": { + "revoke_on": "load_bearing_broken", + "degrade_on": "any_stale_or_broken" + }, + "sealed_at": "2026-06-27T14:47:21Z", + "supersedes": null + } +}