From 5fc622c8ca33da5e16e3aa9fbd67ac35dd038393 Mon Sep 17 00:00:00 2001 From: rgrant Date: Mon, 20 Jul 2026 18:44:45 +0000 Subject: [PATCH 01/10] 20260720 inital issue object-store spec --- .../D24-git-object-store-for-github-issues.md | 603 ++++++++++++++++++ 1 file changed, 603 insertions(+) create mode 100644 docs/rfd/drafts/D24-git-object-store-for-github-issues.md diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md new file mode 100644 index 00000000..529ab043 --- /dev/null +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -0,0 +1,603 @@ +# RFD D24: Git Object Store for GitHub Issues + +- **Status**: Draft +- **Category**: Design +- **Authors**: rgrant +- **Date**: 2026-07-19 + +## Summary + +This RFD describes a local-first store for GitHub issues, kept as git objects +in the project's own `.git` under a dedicated ref namespace. +A scraper tool mirrors issues and their comments from GitHub through +`jp_github`; contributors sync the store with ordinary push and pull. +Each issue is a set of per-writer, append-only operation logs; issue state is +computed deterministically from their union, without merge conflicts and +without dropped data. + +## Motivation + +Contributors need offline access to the project's issues, and later the +ability to edit them locally (`issue append`) and build views over them (a +kanban tool). +The consumers are our own jp-tools: `issue show` first, `issue append` and a +kanban view later. + +GitHub is a read-only interface where some issues get added. +It is not the authority. +The local store is the source of truth, designed from day one for a world +where local edits are made concurrently on machines that do not know about +each other. + +Three storage approaches fail the requirements: + +- **Checked-in files** cause merge conflicts for people working in worktrees, + and dirty the code's evolution with their own commit history. +- **A centralized per-user store** mixing all repositories (as radicle's + `/storage/` does) is rejected for security reasons - it would + require inventing a DSL for access control. +- **Last-write-wins merging** (git-bug's approach, in elaborated form) hides + conflicts by silently dropping the losing write. + +Storing issues as git objects in the repo's own `.git` — objects in the object +database, refs under a dedicated namespace, nothing in the worktree — avoids +all three. +A contributor's responsibility is to pull; after that they have everything +they need for offline work. + +## Design + +### What the user sees + +```sh +# sync: fetch everyone's issue refs, push your own +jp-tools issue sync + +# read a folded issue, offline +jp-tools issue show 42 + +# poke at the raw store with stock git +git for-each-ref refs/jp/issues/42/ +git log refs/jp/issues/42/ +``` + +Daily git work is unaffected: `git log` (HEAD), `git branch`, and default +clones never see the store. +The sync tool configures the `refs/jp/issues/*` fetch refspec. +Documentation explains the git integration using refs, and how to edit +the git config manually. + +### Store layout + +The store lives inside the `.git` of the repository whose issues it holds. +Each issue has one ref per writer: + +``` +refs/jp/issues// +``` + +- `` is the GitHub issue number, or — in the later local-creation + phase — a `jp_id`-formatted id for an issue born locally; the two forms + are syntactically disjoint. + The literal `meta` is reserved for the store-level metadata log. + The store holds the repo's own issues only; a cross-repo mirror would get a + sibling namespace and is out of scope. +- `` identifies one worktree of one clone, held by one signer. + It is composed of three segments, + `/-/`: + the workspace id scopes the writer to the JP workspace that produced it; + the key hash binds the ref to a signing key (the fold rejects commits + under a key-hash segment that are not signed by the matching key); the + avatar nickname is a human-readable label with no authority; the worktree + id is a `jp_id`-formatted id minted per worktree on first write, stored in + that worktree's own state under `.git/`, never checked in and never + synced. + Every worktree of every clone is its own writer. + +A ref is only ever written by its owner — one worktree — so no two +replicas ever contend on a ref. +Every push is a fast-forward, and no merge commit exists anywhere in the +store. + +### What each commit contains + +Each write is one standard commit object, used as an envelope: + +- **tree**: a single blob, `ops.json` — the payload. + It carries the operations of this write, a `seen_heads` field (defined + below), and a format version. + Tools read and write `ops.json` and nothing else; its versioned schema is + the compatibility boundary between replicas, and changing it is a breaking + change for every clone that holds copies. +- **parent**: the previous commit on this writer's chain (none for the first). + Exactly one parent, always: every ref is a strictly linear chain. +- **author/committer**: the writer id as name, write timestamp. +- **message**: a one-line summary generated by the tool, e.g. + `observe #42: set-state closed, add-comment 2140518390`. + No human ever writes one; it exists so `git log` stays legible when + debugging the store. + Never parsed. + +Causality between writers is recorded in the `ops.json` payload; commits +never have multiple parents. +`seen_heads` maps each foreign writer id to the newest op `id` this writer +had incorporated from that chain at write time. +The resolution protocol built on it is specified in the section "Computing +issue state". +Acknowledgments double as tamper evidence: a writer who rewrote history to +drop ops would leave other writers' `seen_heads` pointing at op ids that no +longer exist. + +### Operations + +Operations are fine-grained, one vocabulary for scraped and (later) local +writes. +Each op carries a stable `jp_id`-formatted `id`, minted at creation; the id +is what tombstones and redaction stubs reference. + +| Op | Target | Fold semantics | +| --- | --- | --- | +| `set-title` | issue | multi-value register | +| `set-body` | issue | multi-value register | +| `set-state` | issue | multi-value register | +| `add-label` / `remove-label` | issue | observed-remove set | +| `add-comment` | comment (by GitHub id) | grow-only set | +| `set-comment-body` | comment (by GitHub id) | multi-value register | +| `set-priority` | issue | multi-value register | +| `delete-issue` | issue | tombstone | +| `delete-comment` | comment (by GitHub id) | tombstone | +| `redact-op` | op (by op id) | tombstone | +| `keep` | issue or comment | keep (defined below) | +| `set-allowed-labels` | store metadata | multi-value register | + +The three fold semantics named in the table are defined in the section +"Computing issue state", after the causal order they depend on. + +### Computing issue state + +Issue state is computed by a *fold*: collecting the ops from every writer's +log and applying them in causal order. + +**Causal order.** *Happens-before* is the smallest transitive relation built +from two edge kinds: + +1. Chain order: an op happens-before every later op in its own chain. +2. Acknowledgment: when op C's `seen_heads` maps writer W to op id X, then X + and every op before X in W's chain happen-before C. + +Two ops are **concurrent** when neither happens-before the other. +This relation is the only input to conflict resolution. + +**Resolution**, per the fold semantics in the op table: + +- **Multi-value register** (title, body, state, comment body, priority): the + register's state is the set of ops on it that do not happen-before another + op on the same register. + One op in that set yields a single value — the normal case. + Two or more ops in that set — possible only for concurrent writes — are + all retained and all displayed, ordered by op id. +- **Observed-remove set** (labels): a label is present when some `add-label` + op for it does not happen-before a `remove-label` op for it. + A remove affects only adds it acknowledged; a concurrent add survives. +- **Grow-only set** (comments): the union of `add-comment` ops. +- **Tombstone** (deletes, redactions): once present, every fold hides the + target from that point on; concurrent edits to the target stay in the log + but are not displayed. + Hiding is the entire mechanism; the bytes are never removed from the + store. +- **Keep** (moderation): a `keep` op from any single moderator (1-of-n) + overrides a tombstone on the same target. + The kept content renders however the keeping moderator chose: full text, + a marker, or a moderator-written summary. + Without a `keep`, a single redact or delete suffices to hide (also + 1-of-n). + +**Procedure:** + +1. Enumerate heads: `git for-each-ref refs/jp/issues//`. +2. Walk each linear chain, collect ops. +3. Compute happens-before; apply the resolution rules. + +The fold is order-independent across replicas: any two clones that have seen +the same commits compute the same state, regardless of how or in what order +the commits arrived. + +### Sync + +`issue sync` is fetch plus push, nothing else: + +1. Fetch `refs/jp/issues/*` with a non-forcing refspec: every chain is + append-only, so every legitimate update is a fast-forward. + A non-fast-forward foreign ref means someone rewrote history; the tool + refuses the update, keeps its local copy, and reports the ref. +2. Push the local writer's refs. + +> [!CAUTION] +> A forcing refspec — the `+` prefix, as in `+refs/jp/issues/*` — disables +> git's fast-forward refusal, which is the store's integrity guard. +> A remote configured with a forcing refspec over `refs/jp/` lets a hostile +> or compromised peer silently overwrite good refs with rewritten history. +> `issue sync` always passes its own explicit non-forcing refspec on the +> `git fetch` command line, which overrides whatever refspecs the remote's +> config carries; it also refuses to run while any remote's configured +> `fetch` entry carries a forcing refspec covering `refs/jp/`. +> Never configure one manually. + +No push race exists: every worktree is its own writer, so no two replicas +ever update the same ref. +Within one worktree, concurrent tool invocations serialize through git's own +ref locking — `update-ref` with the expected old value; on failure, re-read +and retry the append. +No legitimate non-fast-forward ref update exists anywhere in the system. + +### The scraper + +The scraper is one writer among several, recording observations from GitHub's +read-only interface. +Per issue it folds current local state, diffs the scraped issue and comments +against it, and emits ops only for fields that differ. +An unchanged issue writes nothing. +Commits carry provenance: `scraped_at` and the GitHub `updated_at` observed. + +Phase 1 scrapes issues and their conversation comments (comment bodies are +editable on GitHub, hence `set-comment-body`). + +The scraper also detects upstream deletions. +Each run enumerates the full issue list, and the comment id set of each +changed issue; a previously observed issue or comment that is gone (API +404/410, or missing from its id set) gets a `delete-issue` or +`delete-comment` op carrying scraper provenance. + +### Signed commits + +Every writer signs its commits (`git commit-tree -S`; SSH-key signing via +`gpg.format=ssh` keeps the requirement to a key writers already have). +The scraper signs like any other writer. + +Enforcement happens at read time, inside the fold — commits can arrive from +any remote or bundle, so no single point exists through which all writes +pass, and GitHub's signed-commit protections only cover branches. +The fold verifies each commit (`git verify-commit`); ops from unverifiable +commits are excluded from the computed state and surfaced as a warning. +A configuration option (on by default) escalates the warning to a hard failure. + +The trust anchor stays **outside the repository**: a per-user allowed-signers +file (git's `gpg.ssh.allowedSignersFile` format) under the user's own +configuration, with keys exchanged out-of-band. +A checked-in list would let anyone who can push code appoint signers — the +artifact being verified must not control its own trust anchor. + +The signing key is the authoritative identity, and the writer id embeds its +hash: the fold rejects commits that live under a key-hash segment but are +not signed by the matching key. +All worktree refs under the same key hash belong to the same principal. + +### Deletion, redaction, and capabilities + +The store is append-only: no ref is ever rewritten and no object is ever +removed. +Every replica refuses a non-fast-forward update of a foreign ref, so an +author cannot hide a history rewrite — existing replicas detect it +directly, and dangling `seen_heads` references expose it even to fresh +clones. + +`delete-issue`, `delete-comment`, and `redact-op` are ordinary ops: they +propagate like any other, and every fold hides the target from that point +on. +The hidden bytes remain in every clone, permanently. + +**Keeping deleted content.** A user's delete is not final: any single +moderator may keep the deleted messages by recording a `keep` op in their +own chain, naming the tombstoned target and the rendering they chose — +full text, a marker, or a moderator-written summary. +Nothing needs rescuing: the content is still in the author's chain, merely +hidden, and the `keep` op changes how the fold renders it. + +**Capabilities.** Actions are authorized per signing key, evaluated at fold +time. +The capability policy lives with the allowed-signers file, outside the +repository, consistent with the trust anchor decision. +Ops signed by a key lacking the required capability are excluded from the +computed state and surfaced, with the same warn/enforce handling as +unverifiable signatures. +The originator of an issue or comment is the key that signed its creating +op. + +| Action | Op | Default capability | +| --- | --- | --- | +| close / reopen issue | `set-state` | any trusted writer | +| tag / untag (incl. `wontfix`) | `add-label` / `remove-label` | any trusted writer | +| reprioritize (kanban ordering) | `set-priority` | any trusted writer | +| edit allowed tags | `set-allowed-labels` | moderators | +| moderate issue / comment | `redact-op` | any single moderator (1-of-n) | +| keep deleted content | `keep` | any single moderator (1-of-n) | +| delete issue | `delete-issue` | originator while no other writer has appended activity; moderators otherwise | +| delete comment | `delete-comment` | originator, subject to moderator keep; moderators | + +Store-level metadata (the allowed-labels vocabulary) lives in its own log at +`refs/jp/issues/meta/`, using the same op machinery as an issue. + +### Implementation: git plumbing subprocesses + +All object and ref access shells out to the `git` binary through the existing +`ProcessRunner` abstraction in the tools crate: + +- writes: `git hash-object -w --stdin`, `git mktree`, `git commit-tree`, + `git update-ref` +- reads: `git for-each-ref`, `git rev-list`, one-shot + `git cat-file --batch` with all requests written to stdin upfront + +Rationale, in order of weight: + +1. **Coexistence is correctness-critical.** The store lives inside + repositories people care about. + The git binary can never disagree with itself about locking, gc, packfile + formats, or the repo's object format (SHA-256 repos are inherited for + free). +2. **Scale does not justify a library.** Hundreds of issues, dozens of ops + each; incremental scrapes write a handful of commits. + The initial import is a one-time bulk write of a few thousand spawns. +3. **Zero new dependencies** in a project that runs cargo-vet, and the + subprocess pattern — including `MockProcessRunner` tests and real-git + integration tests — already exists in the tools crate. + +This decision has a pre-agreed revision trigger: if computing state across +all issues (the kanban view) measures slow, the read path moves to the `gix` +crate (reading is its most mature half) while writes stay as plumbing. +The on-disk format is git's either way, so stored data does not change. + +## Drawbacks + +- Refs grow with issues × worktrees and are permanent: a chain's ops are + part of issue state, so refs cannot be pruned. +- The store only grows. + Deleted and redacted content still occupies space in every clone forever. +- Reads assemble N writer heads per issue instead of walking one DAG, and + cross-writer causality is invisible to `git log --graph` — only the fold + can reconstruct it. + Acceptable: the consumers are exclusively our own tools. +- Ops carry a small map of writer id to op id (the `seen_heads` field). +- Every writer must have commit signing configured, including scraper + automation. + + [rgrant 20260720 00:00 UTC] well, verification can be optional. + is there a config.toml for this? + if so, add a VerificationRequired= field. + everything on and after that commit requires verification. + +- Subprocess-based reads put a performance ceiling on computing state over + many issues; the Implementation section names the measured trigger for + moving reads to `gix`. + +## Alternatives + +- **Checked-in files** (e.g. `issues/*.json` in the worktree): merge + conflicts in worktrees, issue churn pollutes code history. + Rejected in Motivation. +- **A dedicated bare repository** owned by JP (or a radicle-style centralized + store): breaks "pull and you have everything", and centralizing many + repositories in one store is rejected for security reasons. +- **State snapshots instead of op logs**: snapshot merges have no principled + answer to concurrent divergence; the format is a distributed contract that + every collaborator's clone holds copies of, so migrating later means a + coordinated flag-day. + Op-log from day one. +- **Coarse `observe` ops** (full issue JSON per write): simpler to write, but + pushes interpretation into the fold and makes local edits a second, + differently-shaped op family. + Fine-grained ops keep `issue append` symmetrical. +- **One shared ref per issue, merge-on-push**: with N replicas syncing + pairwise at arbitrary times, shared-ref convergence mints bookkeeping merge + commits at every divergent sync, and independent joins of the same heads + themselves diverge. + Per-writer refs eliminate the entire category. +- **Causality as commit parents** (multi-parent commits referencing foreign + heads): structurally merge commits, which this design forbids; the + `seen_heads` field carries the same information while keeping every chain + linear. +- **git-bug / git-appraise**: closest prior art, same refs-in-repo approach, + but git-bug's elaborated last-write-wins hides conflicts that drop data. +- **`gix` or `git2` instead of plumbing subprocesses**: see the rationale + table in Design; a large vet surface (`gix`) or a C dependency (`git2`) + buys speed the workload does not need, at coexistence risk the store cannot + afford. +- **`git fast-import` for bulk writes**: a second command language to + generate and debug; the write volume does not demand it. + Reach for it only if initial import time annoys someone. + +## Non-Goals + +- Pull requests, review comments, and reactions. +- Local issue creation and editing (`issue append`). + The op vocabulary and store format are built for it, but the write path is + a later phase. +- The kanban tool and any state caching for it (the `set-priority` op is + registered here; the tool that consumes it is not). +- Cross-repo mirroring. +- Moderation governance beyond the 1-of-n rules: vote thresholds, disputes + between moderators, appeals. +- Physical removal of store content: deletion only hides. + +## Risks and Open Questions + +- **Deleted content persists in every clone.** Every clone permanently + holds every op ever synced, including content its author deleted and + content a moderator redacted. + This is deliberate — the store is append-only — but it means true + erasure (leaked credentials, legal demands) is impossible inside the + system. + The remedy for a leaked secret is rotating the secret. +- **Capability policy is per-user.** Like the allowed-signers file, the + capability policy lives outside the repository, so two users can compute + different folded states from the same commits. + Tools must surface excluded ops, so the divergence stays visible. +- **Large payloads.** When `issue append` needs content too large for + `ops.json` (logs, screenshots), the payload goes to the `.jp/blobs/` store + of [RFD 066], referenced from the op by SHA-256. + The signed op carries the checksum, so signature verification extends to + the blob content. + Consequence to accept: blobs travel with ordinary worktree commits, not + with `issue sync` ref exchange, so an op can reference a blob its reader + has not yet pulled. + Details deferred to the append-phase RFD. + +- **Verification cost.** `git verify-commit` per commit at read time is + subprocess-heavy; verification results may need caching. + Measure before optimizing. + +## Continuing Questions + +These questions came out of analyzing one concrete attack, and each needs a +decision before the design is complete. + +The attack scenario: Mallory is a developer with a git remote she controls. +She rewrites the history of one writer chain under `refs/jp/issues/42/`, +removing the commit that holds the plan for issue 42. +Alice fetches code and issue refs directly from Mallory's remote. +Bob runs the git hosting server that the team otherwise shares, and Alice +has push access to it. + +The analysis showed: if Alice already holds the current value of the +rewritten ref, her non-forcing fetch refuses the update (a chain missing a +commit fails to fast-forward). +If Alice is behind, or fetching these refs for the first time, she accepts +Mallory's version — a first fetch has no prior value to compare against, +and signatures authenticate authorship of the commits that are present +without proving that the set is complete. +Alice's replica is then corrupted until the evidence described in question 3 +surfaces, and her next fetch from an honest replica jams on that ref. +Spreading the corruption further requires a force push to a server that +accepts one, which motivates question 1. + +1. **Server-side receive gate.** A git hosting server that accepts pushes + holds a replica of the store, but plain git applies no special rules to + `refs/jp/issues/*`: a pusher with write access can force-push a rewritten + chain, and the server accepts what every jp tool would refuse. + Self-managed hosts (GitLab server hooks, plain git `pre-receive`) can run + a hook that rejects any update to `refs/jp/issues/*` that fails to + fast-forward the ref's current value, contains a commit with more than + one parent, or contains a commit whose signature does not match the key + hash in the ref path — the same checks the fold applies at read time, + run at push time. + + Question: does this RFD ship that hook, require it, or document it as + optional hardening? + + [rgrant 20260720 11:54 UTC] no RFD ships anything. it's a fucking + discussion. document the attack, the solution plan, and what + tools will be used to complete that phase of the plan. + + Hosted platforms without custom hooks cannot run it; what is the stated + posture for them? + + [rgrant 20260720 11:53 UTC] is this one question or two, asshole. the attack + is not about hosted platforms. if Bob's git tooling has a + pre-receive hook then so does any platform. + + focus only on Alice, Bob, and Mallory with their git remotes to + each other and their git tools such as pre-fetch. focus on what + otherwise-normal git tool usage exposes the attack. + +2. **Local ref journal.** A user can bypass `issue sync` by running `git + fetch` by hand with a forcing refspec (see the caution block in the + section "Sync"), silently + replacing good refs with rewritten ones — the tool never runs, so it + cannot object. + Setting `core.logAllRefUpdates=always` in the repository makes git + journal every ref update, including updates under `refs/jp/`, into + reflogs that record each transition's old and new values and that protect + the old commits from garbage collection while the entries live. + `issue sync` could read that journal at the start of every run, flag any + transition that failed to fast-forward, and offer to restore the + journaled prior value. + Question: does the tool set this configuration automatically and treat + the journal check as a standard part of every sync? + + [rgrant 20260720 12:00 UTC] yes, while informing the user IF it is a change + in git configuration. + +3. **Missing-acknowledgment detection as a mandatory rule.** Each op's + `seen_heads` field names, per foreign writer, the newest op id the author + had incorporated (see the section "What each commit contains"). + A chain rewritten to drop an op leaves other writers' `seen_heads` + entries naming an op id that no longer exists anywhere in the store. + That evidence is available to every replica, including a fresh clone that + has no prior refs to compare against — for a fresh clone it is the only + rewrite detection there is. + The design currently mentions this evidence in one sentence in the + section "What each commit contains". + Question: promote it to a mandatory fold rule? + + [rgrant 20260720 12:09 UTC] are you fucking asking me whether to catch a bug? + yes. catch the bug. + + And what is the failure mode when it fires — warn and render what + remains, or refuse to render the issue until a human adjudicates which + chain is authentic? + + [rgrant 20260720 12:09 UTC] specific typed error that stops the program. + run --fix-interactive to resolve. + offer to drop the offending ref. + +4. **Withholding.** A remote can serve truthful but stale refs: every commit + validly signed, every update a fast-forward, and the newest ops simply + absent. + A reader served only by that remote sees an issue frozen in the past. + No mechanical check can distinguish withholding from ordinary propagation + delay, and the gap heals on the next sync with any replica that has the + newer state. + Question: accept this as an inherent limit and record it under Risks, or + mitigate by fetching from more than one remote and raising an alarm when + two remotes disagree about the same writer ref? + + [rgrant 20260720 12:11 UTC] simply update to the correct tip when you see + it. do not penalize sync from participants with older tips, + but do not "update" to older state. leave fetch decisions to + user. + +## Implementation Plan + +Each phase is independently reviewable and mergeable. + +1. **Store primitives** in the tools crate: writer-id minting and storage, + `ops.json` schema (versioned), commit read/write via `ProcessRunner` + plumbing, ref enumeration. + Unit-tested against `MockProcessRunner`, integration-tested against real + temp repos. +2. **State computation (the fold)**: chain walking, causal ordering from + `seen_heads`, deterministic tiebreak, per-field semantics (multi-value + registers, observed-remove set, grow-only set). + Pure logic over data fetched by phase 1; property-style tests for + order-independence. +3. **Scraper** (`issue sync`, write side): scrape via `jp_github`, diff + scraped state against computed local state, emit ops — including + `delete-issue` / `delete-comment` tombstones for upstream deletions — + and sign commits. + Depends on phases 1–2. +4. **Sync** (`issue sync`, transport side): fetch refspec configuration, + fast-forward pushes, refusal to run while any remote's configured + `fetch` entry carries a forcing refspec covering `refs/jp/`. +5. **`issue show`**: compute and render one issue's state, including + surfaced multi-value conflicts and unverified-writer warnings. +6. **Signature verification** during state computation: `verify-commit` + against the per-user allowed-signers file, with the warn/enforce + configuration option. +7. **Deletion and keeps** (`issue delete`): tombstone ops, `keep` ops, + capability checks against the per-user policy. + +## References + +- [RFD 066] — Content-Addressable Blob Store: content-addressed storage for + conversation blobs, and the designated home for large payloads in the + later `issue append` phase (see the section "Risks and Open Questions"). + +- [git-bug] — issues as git objects in refs, closest prior art. +- [git-appraise] — code review as git objects in refs. +- [radicle COBs] — collaborative objects as commit DAGs; this design borrows + the op-log idea but rejects centralized per-user storage and multi-parent + causality. + +[RFD 066]: ../066-content-addressable-blob-store.md +[git-bug]: https://github.com/git-bug/git-bug +[git-appraise]: https://github.com/google/git-appraise +[radicle COBs]: https://radicle.xyz/guides/protocol#collaborative-objects From 08325cfa966b5fa8e620028b4aa9a77d1076a1e5 Mon Sep 17 00:00:00 2001 From: rgrant Date: Tue, 21 Jul 2026 01:45:05 +0000 Subject: [PATCH 02/10] 20260721 issue storage improvements --- .../D24-git-object-store-for-github-issues.md | 225 +++++++++--------- 1 file changed, 111 insertions(+), 114 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index 529ab043..06ae894d 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -127,6 +127,8 @@ issue state". Acknowledgments double as tamper evidence: a writer who rewrote history to drop ops would leave other writers' `seen_heads` pointing at op ids that no longer exist. +The fold checks for exactly this dangling reference; see the section +"Computing issue state". ### Operations @@ -192,11 +194,22 @@ This relation is the only input to conflict resolution. Without a `keep`, a single redact or delete suffices to hide (also 1-of-n). +**Missing acknowledgments.** Before applying the resolution rules, the fold +checks every `seen_heads` reference: an entry naming an op id that exists +nowhere in the store is evidence of a rewritten chain (see the section +"Attack Analysis: Chain Rewrite"). +The fold stops with a specific typed error instead of rendering partial +state; running with `--fix-interactive` walks through resolution, including +the option to drop the offending ref. +For a fresh clone, which has no prior ref values to compare against, this +check is the only rewrite detection there is. + **Procedure:** 1. Enumerate heads: `git for-each-ref refs/jp/issues//`. 2. Walk each linear chain, collect ops. -3. Compute happens-before; apply the resolution rules. +3. Run the missing-acknowledgment check; compute happens-before; apply the + resolution rules. The fold is order-independent across replicas: any two clones that have seen the same commits compute the same state, regardless of how or in what order @@ -223,6 +236,24 @@ the commits arrived. > `fetch` entry carries a forcing refspec covering `refs/jp/`. > Never configure one manually. +`issue sync` also maintains a local ref journal as a second guard. +It sets `core.logAllRefUpdates=always` in the repository — informing the +user whenever this changes existing git configuration — so git journals +every ref update into reflogs recording each transition's old and new +values, including a hand-run `git fetch` with a forcing refspec that +bypasses the tool entirely. +Every run starts by reading that journal, flags any `refs/jp/` transition +that failed to fast-forward, and offers to restore the journaled prior +value. +The reflog entries also protect the overwritten commits from garbage +collection while they live, so the restore is always possible. + +A remote serving older tips is harmless. +Refs never move backward — an older tip fails to fast-forward — and the +newer state is adopted on the next sync with any replica that has it. +Syncing with a participant who is behind carries no penalty, and which +remotes to fetch from stays the user's decision. + No push race exists: every worktree is its own writer, so no two replicas ever update the same ref. Within one worktree, concurrent tool invocations serialize through git's own @@ -230,6 +261,20 @@ ref locking — `update-ref` with the expected old value; on failure, re-read and retry the append. No legitimate non-fast-forward ref update exists anywhere in the system. +### Server-side receive gate + +A git server that accepts pushes holds a replica of the store, but plain +git applies no special rules to `refs/jp/issues/*`: a pusher with write +access can force-push a rewritten chain, and the server accepts what every +jp tool would refuse. +A `pre-receive` hook closes the gap by rejecting any update to +`refs/jp/issues/*` that fails to fast-forward the ref's current value, +contains a commit with more than one parent, or contains a commit whose +signature does not match the key hash in the ref path — the same checks +the fold applies at read time, run at push time. +The hook is stock git tooling: any server whose operator can install +`pre-receive` hooks can run it. + ### The scraper The scraper is one writer among several, recording observations from GitHub's @@ -261,6 +306,15 @@ The fold verifies each commit (`git verify-commit`); ops from unverifiable commits are excluded from the computed state and surfaced as a warning. A configuration option (on by default) escalates the warning to a hard failure. +Verification is required only from a configured cutoff. +A `verification_required = ""` configuration field names a +commit: that commit and everything after it (in the causal order of the +section "Computing issue state") require verification, while commits before +the cutoff are exempt from the checks above. +With the field unset, no commit requires verification. +This lets a store adopt signing late without invalidating its earlier +history. + The trust anchor stays **outside the repository**: a per-user allowed-signers file (git's `gpg.ssh.allowedSignersFile` format) under the user's own configuration, with keys exchanged out-of-band. @@ -357,14 +411,8 @@ The on-disk format is git's either way, so stored data does not change. can reconstruct it. Acceptable: the consumers are exclusively our own tools. - Ops carry a small map of writer id to op id (the `seen_heads` field). -- Every writer must have commit signing configured, including scraper - automation. - - [rgrant 20260720 00:00 UTC] well, verification can be optional. - is there a config.toml for this? - if so, add a VerificationRequired= field. - everything on and after that commit requires verification. - +- Every writer whose commits fall under the `verification_required` cutoff + must have commit signing configured, including scraper automation. - Subprocess-based reads put a performance ceiling on computing state over many issues; the Implementation section names the measured trigger for moving reads to `gix`. @@ -418,7 +466,7 @@ The on-disk format is git's either way, so stored data does not change. between moderators, appeals. - Physical removal of store content: deletion only hides. -## Risks and Open Questions +## Risks - **Deleted content persists in every clone.** Every clone permanently holds every op ever synced, including content its author deleted and @@ -441,119 +489,65 @@ The on-disk format is git's either way, so stored data does not change. has not yet pulled. Details deferred to the append-phase RFD. +- **Withholding is undetectable.** A remote can serve truthful but stale + refs: every commit validly signed, every update a fast-forward, and the + newest ops simply absent. + A reader served only by that remote sees an issue frozen in the past, and + no mechanical check distinguishes withholding from ordinary propagation + delay. + Accepted as inherent: refs never move backward, the newer tip is adopted + as soon as any replica that has it is fetched from, and the choice of + remotes is the user's. - **Verification cost.** `git verify-commit` per commit at read time is subprocess-heavy; verification results may need caching. Measure before optimizing. -## Continuing Questions +## Attack Analysis: Chain Rewrite -These questions came out of analyzing one concrete attack, and each needs a -decision before the design is complete. +Several of the mechanisms above — the ref journal, the receive gate, the +missing-acknowledgment rule — exist because of one concrete attack. +This section documents it and maps each defense to the design mechanism +that closes it. -The attack scenario: Mallory is a developer with a git remote she controls. +Mallory is a developer with a git remote she controls. She rewrites the history of one writer chain under `refs/jp/issues/42/`, removing the commit that holds the plan for issue 42. Alice fetches code and issue refs directly from Mallory's remote. -Bob runs the git hosting server that the team otherwise shares, and Alice -has push access to it. +Bob runs the git server that the team otherwise shares, and Alice has push +access to it. -The analysis showed: if Alice already holds the current value of the -rewritten ref, her non-forcing fetch refuses the update (a chain missing a -commit fails to fast-forward). +If Alice already holds the current value of the rewritten ref, her +non-forcing fetch refuses the update: a chain missing a commit fails to +fast-forward. If Alice is behind, or fetching these refs for the first time, she accepts Mallory's version — a first fetch has no prior value to compare against, and signatures authenticate authorship of the commits that are present without proving that the set is complete. -Alice's replica is then corrupted until the evidence described in question 3 -surfaces, and her next fetch from an honest replica jams on that ref. -Spreading the corruption further requires a force push to a server that -accepts one, which motivates question 1. - -1. **Server-side receive gate.** A git hosting server that accepts pushes - holds a replica of the store, but plain git applies no special rules to - `refs/jp/issues/*`: a pusher with write access can force-push a rewritten - chain, and the server accepts what every jp tool would refuse. - Self-managed hosts (GitLab server hooks, plain git `pre-receive`) can run - a hook that rejects any update to `refs/jp/issues/*` that fails to - fast-forward the ref's current value, contains a commit with more than - one parent, or contains a commit whose signature does not match the key - hash in the ref path — the same checks the fold applies at read time, - run at push time. - - Question: does this RFD ship that hook, require it, or document it as - optional hardening? - - [rgrant 20260720 11:54 UTC] no RFD ships anything. it's a fucking - discussion. document the attack, the solution plan, and what - tools will be used to complete that phase of the plan. - - Hosted platforms without custom hooks cannot run it; what is the stated - posture for them? - - [rgrant 20260720 11:53 UTC] is this one question or two, asshole. the attack - is not about hosted platforms. if Bob's git tooling has a - pre-receive hook then so does any platform. - - focus only on Alice, Bob, and Mallory with their git remotes to - each other and their git tools such as pre-fetch. focus on what - otherwise-normal git tool usage exposes the attack. - -2. **Local ref journal.** A user can bypass `issue sync` by running `git - fetch` by hand with a forcing refspec (see the caution block in the - section "Sync"), silently - replacing good refs with rewritten ones — the tool never runs, so it - cannot object. - Setting `core.logAllRefUpdates=always` in the repository makes git - journal every ref update, including updates under `refs/jp/`, into - reflogs that record each transition's old and new values and that protect - the old commits from garbage collection while the entries live. - `issue sync` could read that journal at the start of every run, flag any - transition that failed to fast-forward, and offer to restore the - journaled prior value. - Question: does the tool set this configuration automatically and treat - the journal check as a standard part of every sync? - - [rgrant 20260720 12:00 UTC] yes, while informing the user IF it is a change - in git configuration. - -3. **Missing-acknowledgment detection as a mandatory rule.** Each op's - `seen_heads` field names, per foreign writer, the newest op id the author - had incorporated (see the section "What each commit contains"). - A chain rewritten to drop an op leaves other writers' `seen_heads` - entries naming an op id that no longer exists anywhere in the store. - That evidence is available to every replica, including a fresh clone that - has no prior refs to compare against — for a fresh clone it is the only - rewrite detection there is. - The design currently mentions this evidence in one sentence in the - section "What each commit contains". - Question: promote it to a mandatory fold rule? - - [rgrant 20260720 12:09 UTC] are you fucking asking me whether to catch a bug? - yes. catch the bug. - - And what is the failure mode when it fires — warn and render what - remains, or refuse to render the issue until a human adjudicates which - chain is authentic? - - [rgrant 20260720 12:09 UTC] specific typed error that stops the program. - run --fix-interactive to resolve. - offer to drop the offending ref. - -4. **Withholding.** A remote can serve truthful but stale refs: every commit - validly signed, every update a fast-forward, and the newest ops simply - absent. - A reader served only by that remote sees an issue frozen in the past. - No mechanical check can distinguish withholding from ordinary propagation - delay, and the gap heals on the next sync with any replica that has the - newer state. - Question: accept this as an inherent limit and record it under Risks, or - mitigate by fetching from more than one remote and raising an alarm when - two remotes disagree about the same writer ref? - - [rgrant 20260720 12:11 UTC] simply update to the correct tip when you see - it. do not penalize sync from participants with older tips, - but do not "update" to older state. leave fetch decisions to - user. + +Otherwise-normal git tooling exposes the attack at every point where it +would otherwise take hold or spread: + +- **Fast-forward refusal** (the section "Sync"): every replica that already + holds the honest ref refuses Mallory's rewrite outright. +- **The local ref journal** (the section "Sync"): if Alice bypasses the + tool with a hand-run forcing fetch, the reflog records the + non-fast-forward transition; the next `issue sync` flags it and offers to + restore the journaled prior value. +- **Missing-acknowledgment detection** (the section "Computing issue + state"): other writers' `seen_heads` still name the dropped op, so even a + fresh clone — with no prior refs to compare against — detects the + rewrite; the fold stops with a typed error and `--fix-interactive` offers + to drop the offending ref. +- **The server-side receive gate** (the section "Server-side receive + gate"): spreading the corruption through the shared server requires a + force push that Bob's `pre-receive` hook refuses. + +A remote can also *withhold*: serve truthful but stale refs, every commit +validly signed and every update a fast-forward, with the newest ops simply +absent. +That is not a rewrite and no defense above fires; it is an accepted limit, +recorded under Risks, and the sync rules in the section "Sync" guarantee +the gap heals on the next sync with any replica that has the newer state. ## Implementation Plan @@ -565,8 +559,9 @@ Each phase is independently reviewable and mergeable. Unit-tested against `MockProcessRunner`, integration-tested against real temp repos. 2. **State computation (the fold)**: chain walking, causal ordering from - `seen_heads`, deterministic tiebreak, per-field semantics (multi-value - registers, observed-remove set, grow-only set). + `seen_heads`, the missing-acknowledgment check with its typed error and + `--fix-interactive` resolution, deterministic tiebreak, per-field + semantics (multi-value registers, observed-remove set, grow-only set). Pure logic over data fetched by phase 1; property-style tests for order-independence. 3. **Scraper** (`issue sync`, write side): scrape via `jp_github`, diff @@ -576,12 +571,14 @@ Each phase is independently reviewable and mergeable. Depends on phases 1–2. 4. **Sync** (`issue sync`, transport side): fetch refspec configuration, fast-forward pushes, refusal to run while any remote's configured - `fetch` entry carries a forcing refspec covering `refs/jp/`. + `fetch` entry carries a forcing refspec covering `refs/jp/`, + `core.logAllRefUpdates=always` setup and the ref-journal check, and a + reference `pre-receive` hook for server operators. 5. **`issue show`**: compute and render one issue's state, including surfaced multi-value conflicts and unverified-writer warnings. 6. **Signature verification** during state computation: `verify-commit` against the per-user allowed-signers file, with the warn/enforce - configuration option. + configuration option and the `verification_required` cutoff. 7. **Deletion and keeps** (`issue delete`): tombstone ops, `keep` ops, capability checks against the per-user policy. @@ -589,7 +586,7 @@ Each phase is independently reviewable and mergeable. - [RFD 066] — Content-Addressable Blob Store: content-addressed storage for conversation blobs, and the designated home for large payloads in the - later `issue append` phase (see the section "Risks and Open Questions"). + later `issue append` phase (see the section "Risks"). - [git-bug] — issues as git objects in refs, closest prior art. - [git-appraise] — code review as git objects in refs. From 632b70962674a4b37497c68847d03e3a422f6bda Mon Sep 17 00:00:00 2001 From: rgrant Date: Tue, 21 Jul 2026 02:05:44 +0000 Subject: [PATCH 03/10] 20260721 add comment visibility, clarify need for capabilites DSL, clarify upstream deleted issue or comment. --- .../D24-git-object-store-for-github-issues.md | 41 +++++++++++++++---- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index 06ae894d..e926133c 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -145,6 +145,7 @@ is what tombstones and redaction stubs reference. | `add-label` / `remove-label` | issue | observed-remove set | | `add-comment` | comment (by GitHub id) | grow-only set | | `set-comment-body` | comment (by GitHub id) | multi-value register | +| `set-comment-visibility` | comment (by GitHub id) | multi-value register | | `set-priority` | issue | multi-value register | | `delete-issue` | issue | tombstone | | `delete-comment` | comment (by GitHub id) | tombstone | @@ -155,6 +156,15 @@ is what tombstones and redaction stubs reference. The three fold semantics named in the table are defined in the section "Computing issue state", after the causal order they depend on. +`set-comment-visibility` records comment collapsing — GitHub's +comment-minimization feature — as a register holding `visible` or +`collapsed` with a reason (off-topic, outdated, resolved, spam, ...). +Collapse is display muting, not deletion: the fold renders a collapsed +comment as a marker carrying its reason, and un-collapsing is an ordinary +later write to the same register. +The scraper emits this op when it observes a comment's minimized state +change upstream. + ### Computing issue state Issue state is computed by a *fold*: collecting the ops from every writer's @@ -287,11 +297,21 @@ Commits carry provenance: `scraped_at` and the GitHub `updated_at` observed. Phase 1 scrapes issues and their conversation comments (comment bodies are editable on GitHub, hence `set-comment-body`). -The scraper also detects upstream deletions. -Each run enumerates the full issue list, and the comment id set of each -changed issue; a previously observed issue or comment that is gone (API -404/410, or missing from its id set) gets a `delete-issue` or -`delete-comment` op carrying scraper provenance. +The scraper also detects upstream deletions, under one rule: a tombstone +requires positive evidence of deletion, never absence from a +possibly-incomplete listing. +Edits are positive observations and are committed incrementally as scraped; +a deletion is an inference from absence, so deletion ops are emitted only +after the enumeration they are inferred from has run to completion, and +only after a direct GitHub API request for the missing item confirms the +deletion (HTTP 404/410). +Concretely: each run enumerates the full issue list, and the comment id set +of each changed issue; a previously observed issue or comment missing from +its completed enumeration is requested individually from the API, and only +an HTTP 404/410 yields the `delete-issue` or `delete-comment` op, carrying +scraper provenance. +A scrape that aborts mid-run (rate limit, network failure) therefore emits +the edits it observed and no deletions. ### Signed commits @@ -351,6 +371,10 @@ hidden, and the `keep` op changes how the fold renders it. time. The capability policy lives with the allowed-signers file, outside the repository, consistent with the trust anchor decision. +The table below lists *defaults*: the policy format expresses other rules — +different thresholds than 1-of-n, per-action writer sets — without any +change to the store format, because capabilities are evaluated at fold time +and never recorded in ops. Ops signed by a key lacking the required capability are excluded from the computed state and surfaced, with the same warn/enforce handling as unverifiable signatures. @@ -363,6 +387,7 @@ op. | tag / untag (incl. `wontfix`) | `add-label` / `remove-label` | any trusted writer | | reprioritize (kanban ordering) | `set-priority` | any trusted writer | | edit allowed tags | `set-allowed-labels` | moderators | +| collapse / un-collapse comment | `set-comment-visibility` | moderators | | moderate issue / comment | `redact-op` | any single moderator (1-of-n) | | keep deleted content | `keep` | any single moderator (1-of-n) | | delete issue | `delete-issue` | originator while no other writer has appended activity; moderators otherwise | @@ -462,8 +487,10 @@ The on-disk format is git's either way, so stored data does not change. - The kanban tool and any state caching for it (the `set-priority` op is registered here; the tool that consumes it is not). - Cross-repo mirroring. -- Moderation governance beyond the 1-of-n rules: vote thresholds, disputes - between moderators, appeals. +- Moderation governance beyond the default capability rules: vote + thresholds, disputes between moderators, appeals. + The policy format is built to express these later; this RFD fixes only + the defaults. - Physical removal of store content: deletion only hides. ## Risks From 72ec57937911cf4336b80338631b2649329024a3 Mon Sep 17 00:00:00 2001 From: rgrant Date: Wed, 22 Jul 2026 04:52:03 +0000 Subject: [PATCH 04/10] 20260722 clarify ref sharing. --- .../D24-git-object-store-for-github-issues.md | 586 +++++++++--------- 1 file changed, 309 insertions(+), 277 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index e926133c..2382cf2e 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -7,50 +7,49 @@ ## Summary -This RFD describes a local-first store for GitHub issues, kept as git objects -in the project's own `.git` under a dedicated ref namespace. +This RFD describes a local-first store for GitHub issues, kept as git objects in +the project's own `.git` under a dedicated ref namespace. A scraper tool mirrors issues and their comments from GitHub through `jp_github`; contributors sync the store with ordinary push and pull. Each issue is a set of per-writer, append-only operation logs; issue state is -computed deterministically from their union, without merge conflicts and -without dropped data. +computed deterministically from their union, without merge conflicts and without +dropped data. ## Motivation -Contributors need offline access to the project's issues, and later the -ability to edit them locally (`issue append`) and build views over them (a -kanban tool). +Contributors need offline access to the project's issues, and later the ability +to edit them locally (`issue append`) and build views over them (a kanban tool). The consumers are our own jp-tools: `issue show` first, `issue append` and a kanban view later. GitHub is a read-only interface where some issues get added. It is not the authority. -The local store is the source of truth, designed from day one for a world -where local edits are made concurrently on machines that do not know about -each other. +The local store is the source of truth, designed from day one for a world where +local edits are made concurrently on machines that do not know about each other. Three storage approaches fail the requirements: - **Checked-in files** cause merge conflicts for people working in worktrees, and dirty the code's evolution with their own commit history. + They also limit the view of open issues available from branches. - **A centralized per-user store** mixing all repositories (as radicle's - `/storage/` does) is rejected for security reasons - it would - require inventing a DSL for access control. + `/storage/` does) is rejected for security reasons - it would require + inventing a DSL for access control. - **Last-write-wins merging** (git-bug's approach, in elaborated form) hides conflicts by silently dropping the losing write. Storing issues as git objects in the repo's own `.git` — objects in the object database, refs under a dedicated namespace, nothing in the worktree — avoids all three. -A contributor's responsibility is to pull; after that they have everything -they need for offline work. +A contributor's responsibility is to pull; after that they have everything they +need for offline work. ## Design ### What the user sees ```sh -# sync: fetch everyone's issue refs, push your own +# sync: fetch everyone's issue refs, push every issue ref you hold jp-tools issue sync # read a folded issue, offline @@ -61,11 +60,11 @@ git for-each-ref refs/jp/issues/42/ git log refs/jp/issues/42/ ``` -Daily git work is unaffected: `git log` (HEAD), `git branch`, and default -clones never see the store. +Daily git work is unaffected: `git log` (HEAD), `git branch`, and default clones +never see the store. The sync tool configures the `refs/jp/issues/*` fetch refspec. -Documentation explains the git integration using refs, and how to edit -the git config manually. +Documentation explains the git integration using refs, and how to edit the git +config manually. ### Store layout @@ -76,66 +75,64 @@ Each issue has one ref per writer: refs/jp/issues// ``` -- `` is the GitHub issue number, or — in the later local-creation - phase — a `jp_id`-formatted id for an issue born locally; the two forms - are syntactically disjoint. +- `` is the GitHub issue number, or — in the later local-creation phase + — a `jp_id`-formatted id for an issue born locally; the two forms are + syntactically disjoint. The literal `meta` is reserved for the store-level metadata log. The store holds the repo's own issues only; a cross-repo mirror would get a sibling namespace and is out of scope. - `` identifies one worktree of one clone, held by one signer. It is composed of three segments, `/-/`: - the workspace id scopes the writer to the JP workspace that produced it; - the key hash binds the ref to a signing key (the fold rejects commits - under a key-hash segment that are not signed by the matching key); the - avatar nickname is a human-readable label with no authority; the worktree - id is a `jp_id`-formatted id minted per worktree on first write, stored in - that worktree's own state under `.git/`, never checked in and never - synced. + the workspace id scopes the writer to the JP workspace that produced it; the + key hash binds the ref to a signing key (the fold rejects commits under a + key-hash segment that are not signed by the matching key); the avatar nickname + is a human-readable label with no authority; the worktree id is a + `jp_id`-formatted id minted per worktree on first write, stored in that + worktree's own state under `.git/`, never checked in and never synced. Every worktree of every clone is its own writer. -A ref is only ever written by its owner — one worktree — so no two -replicas ever contend on a ref. -Every push is a fast-forward, and no merge commit exists anywhere in the -store. +A ref is only ever written by its owner — one worktree — so no two replicas +ever contend on a ref. +Every push is a fast-forward, and no merge commit exists anywhere in the store. ### What each commit contains Each write is one standard commit object, used as an envelope: - **tree**: a single blob, `ops.json` — the payload. - It carries the operations of this write, a `seen_heads` field (defined - below), and a format version. - Tools read and write `ops.json` and nothing else; its versioned schema is - the compatibility boundary between replicas, and changing it is a breaking - change for every clone that holds copies. + It carries the operations of this write, a `seen_heads` field (defined below), + and a format version. + Tools read and write `ops.json` and nothing else; its versioned schema is the + compatibility boundary between replicas, and changing it is a breaking change + for every clone that holds copies. - **parent**: the previous commit on this writer's chain (none for the first). Exactly one parent, always: every ref is a strictly linear chain. - **author/committer**: the writer id as name, write timestamp. -- **message**: a one-line summary generated by the tool, e.g. - `observe #42: set-state closed, add-comment 2140518390`. - No human ever writes one; it exists so `git log` stays legible when - debugging the store. +- **message**: a one-line summary generated by the tool, e.g. `observe #42: + set-state closed, add-comment 2140518390`. + No human ever writes one; it exists so `git log` stays legible when debugging + the store. Never parsed. -Causality between writers is recorded in the `ops.json` payload; commits -never have multiple parents. -`seen_heads` maps each foreign writer id to the newest op `id` this writer -had incorporated from that chain at write time. -The resolution protocol built on it is specified in the section "Computing +Causality between writers is recorded in the `ops.json` payload; commits never +have multiple parents. +`seen_heads` maps each foreign writer id to the newest op `id` this writer had +incorporated from that chain at write time. +The resolution protocol built on it is specified in the section "Computing issue +state". +Acknowledgments double as tamper evidence: a writer who rewrote history to drop +ops would leave other writers' `seen_heads` pointing at op ids that no longer +exist. +The fold checks for exactly this dangling reference; see the section "Computing issue state". -Acknowledgments double as tamper evidence: a writer who rewrote history to -drop ops would leave other writers' `seen_heads` pointing at op ids that no -longer exist. -The fold checks for exactly this dangling reference; see the section -"Computing issue state". ### Operations Operations are fine-grained, one vocabulary for scraped and (later) local writes. -Each op carries a stable `jp_id`-formatted `id`, minted at creation; the id -is what tombstones and redaction stubs reference. +Each op carries a stable `jp_id`-formatted `id`, minted at creation; the id is +what tombstones and redaction stubs reference. | Op | Target | Fold semantics | | --- | --- | --- | @@ -153,29 +150,31 @@ is what tombstones and redaction stubs reference. | `keep` | issue or comment | keep (defined below) | | `set-allowed-labels` | store metadata | multi-value register | +A register is one field of one target: ops of the same type on the same target +address the same register. The three fold semantics named in the table are defined in the section "Computing issue state", after the causal order they depend on. `set-comment-visibility` records comment collapsing — GitHub's -comment-minimization feature — as a register holding `visible` or -`collapsed` with a reason (off-topic, outdated, resolved, spam, ...). -Collapse is display muting, not deletion: the fold renders a collapsed -comment as a marker carrying its reason, and un-collapsing is an ordinary -later write to the same register. -The scraper emits this op when it observes a comment's minimized state -change upstream. +comment-minimization feature — as a register holding `visible` or `collapsed` +with a reason (off-topic, outdated, resolved, spam, ...). +Collapse is display muting, not deletion: the fold renders a collapsed comment +as a marker carrying its reason, and un-collapsing is an ordinary later write to +the same register. +The scraper emits this op when it observes a comment's minimized state change +upstream. ### Computing issue state -Issue state is computed by a *fold*: collecting the ops from every writer's -log and applying them in causal order. +Issue state is computed by a *fold*: collecting the ops from every writer's log +and applying them in causal order. **Causal order.** *Happens-before* is the smallest transitive relation built from two edge kinds: 1. Chain order: an op happens-before every later op in its own chain. -2. Acknowledgment: when op C's `seen_heads` maps writer W to op id X, then X - and every op before X in W's chain happen-before C. +2. Acknowledgment: when op C's `seen_heads` maps writer W to op id X, then X and + every op before X in W's chain happen-before C. Two ops are **concurrent** when neither happens-before the other. This relation is the only input to conflict resolution. @@ -183,36 +182,49 @@ This relation is the only input to conflict resolution. **Resolution**, per the fold semantics in the op table: - **Multi-value register** (title, body, state, comment body, priority): the - register's state is the set of ops on it that do not happen-before another - op on the same register. + register's state is the set of ops on it that do not happen-before another op + on the same register. One op in that set yields a single value — the normal case. - Two or more ops in that set — possible only for concurrent writes — are - all retained and all displayed, ordered by op id. -- **Observed-remove set** (labels): a label is present when some `add-label` - op for it does not happen-before a `remove-label` op for it. + Two or more ops in that set — possible only for concurrent writes — are all + part of the register's state, and `issue show` renders every one, ordered by + op id. + The register holds a single value again only when a later write acknowledges + every op in that set — that write happens-after each of them, and the fold + resolves the register to it. +- **Observed-remove set** (labels): a label is present when some `add-label` op + for it does not happen-before a `remove-label` op for it. A remove affects only adds it acknowledged; a concurrent add survives. - **Grow-only set** (comments): the union of `add-comment` ops. -- **Tombstone** (deletes, redactions): once present, every fold hides the - target from that point on; concurrent edits to the target stay in the log - but are not displayed. - Hiding is the entire mechanism; the bytes are never removed from the - store. +- **Tombstone** (deletes, redactions): once present, every fold hides the target + from that point on; concurrent edits to the target stay in the log but are not + displayed. + Hiding is the entire mechanism; the bytes are never removed from the store. - **Keep** (moderation): a `keep` op from any single moderator (1-of-n) overrides a tombstone on the same target. - The kept content renders however the keeping moderator chose: full text, - a marker, or a moderator-written summary. - Without a `keep`, a single redact or delete suffices to hide (also - 1-of-n). + The kept content renders however the keeping moderator chose: full text, a + marker, or a moderator-written summary. + Without a `keep`, a single redact or delete suffices to hide (also 1-of-n). **Missing acknowledgments.** Before applying the resolution rules, the fold -checks every `seen_heads` reference: an entry naming an op id that exists -nowhere in the store is evidence of a rewritten chain (see the section -"Attack Analysis: Chain Rewrite"). -The fold stops with a specific typed error instead of rendering partial -state; running with `--fix-interactive` walks through resolution, including -the option to drop the offending ref. -For a fresh clone, which has no prior ref values to compare against, this -check is the only rewrite detection there is. +checks every `seen_heads` reference. +A `seen_heads` entry can name an op id that exists nowhere in the local store. +A missing op id has two possible causes: a writer rewrote a chain and dropped +the op, or the local replica has not yet fetched the commits that carry the op. +No local check can tell the two causes apart, because sync is pairwise and +asynchronous: a replica can legitimately receive an acknowledgment of an op +before receiving the op. +The fold stops with a typed error naming the writer and the missing op id, +renders nothing, and tells the user to fetch from more remotes. +Fetching cures the innocent cause. +When fetching from every available remote still leaves the op id missing, or +when a refused non-fast-forward fetch or a ref-journal entry points at a +specific chain, the cause is a rewritten chain (see the section "Attack +Analysis: Chain Rewrite"). +Running with `--fix-interactive` walks through resolution, including the option +to drop the offending ref. +A fresh clone has no prior ref values to compare against, so a fresh clone +detects rewrites only through missing acknowledgments, and a missing +acknowledgment is grounds to investigate, not proof of a rewrite. **Procedure:** @@ -221,9 +233,9 @@ check is the only rewrite detection there is. 3. Run the missing-acknowledgment check; compute happens-before; apply the resolution rules. -The fold is order-independent across replicas: any two clones that have seen -the same commits compute the same state, regardless of how or in what order -the commits arrived. +The fold is order-independent across replicas: any two clones that have seen the +same commits compute the same state, regardless of how or in what order the +commits arrived. ### Sync @@ -233,57 +245,80 @@ the commits arrived. append-only, so every legitimate update is a fast-forward. A non-fast-forward foreign ref means someone rewrote history; the tool refuses the update, keeps its local copy, and reports the ref. -2. Push the local writer's refs. +2. Push every `refs/jp/issues/*` ref the local replica holds: the local writer's + own refs, plus all foreign refs picked up by fetching. + Pushing foreign refs spreads every writer's chain to every remote. + If sync pushed only the local writer's refs, then a remote could be missing + some writer's chain forever, because no contributor is obligated to push + another writer's chain to that remote. + A clone made from a remote that is missing a chain cannot compute issue + state: surviving chains hold `seen_heads` entries that name op ids inside the + missing chain, the fold cannot find those op ids, and the fold stops with an + error. + Pushing foreign refs is safe: each chain has exactly one writer and only + grows, so two replicas pushing the same ref push the same tip, or one tip is + an extension of the other. + When the remote is already ahead on a ref, git rejects the push; the + rejection is harmless, and the next fetch picks up the newer commits. > [!CAUTION] > A forcing refspec — the `+` prefix, as in `+refs/jp/issues/*` — disables > git's fast-forward refusal, which is the store's integrity guard. -> A remote configured with a forcing refspec over `refs/jp/` lets a hostile -> or compromised peer silently overwrite good refs with rewritten history. -> `issue sync` always passes its own explicit non-forcing refspec on the -> `git fetch` command line, which overrides whatever refspecs the remote's -> config carries; it also refuses to run while any remote's configured -> `fetch` entry carries a forcing refspec covering `refs/jp/`. +> A remote configured with a forcing refspec over `refs/jp/` lets a hostile or +> compromised peer silently overwrite good refs with rewritten history. +> `issue sync` always passes its own explicit non-forcing refspec on the `git +> fetch` command line, which overrides whatever refspecs the remote's config +> carries; it also refuses to run while any remote's configured `fetch` entry +> carries a forcing refspec covering `refs/jp/`. > Never configure one manually. `issue sync` also maintains a local ref journal as a second guard. -It sets `core.logAllRefUpdates=always` in the repository — informing the -user whenever this changes existing git configuration — so git journals -every ref update into reflogs recording each transition's old and new -values, including a hand-run `git fetch` with a forcing refspec that -bypasses the tool entirely. -Every run starts by reading that journal, flags any `refs/jp/` transition -that failed to fast-forward, and offers to restore the journaled prior -value. -The reflog entries also protect the overwritten commits from garbage -collection while they live, so the restore is always possible. +It sets `core.logAllRefUpdates=always` in the repository — informing the user +whenever this changes existing git configuration — so git journals every ref +update into reflogs recording each transition's old and new values, including a +hand-run `git fetch` with a forcing refspec that bypasses the tool entirely. +Every run starts by reading that journal, flags any `refs/jp/` transition that +failed to fast-forward, and offers to restore the journaled prior value. +The reflog entries also protect the overwritten commits from garbage collection +while they live, so the restore is always possible. A remote serving older tips is harmless. -Refs never move backward — an older tip fails to fast-forward — and the -newer state is adopted on the next sync with any replica that has it. -Syncing with a participant who is behind carries no penalty, and which -remotes to fetch from stays the user's decision. - -No push race exists: every worktree is its own writer, so no two replicas -ever update the same ref. -Within one worktree, concurrent tool invocations serialize through git's own -ref locking — `update-ref` with the expected old value; on failure, re-read -and retry the append. +Refs never move backward — an older tip fails to fast-forward — and the newer +state is adopted on the next sync with any replica that has it. +Syncing with a participant who is behind carries no penalty, and which remotes +to fetch from stays the user's decision. + +Two replicas may push the same ref, but a push race cannot corrupt a chain: only +the owning worktree ever appends commits to a chain, so competing pushes carry +the same tip, or one tip is an extension of the other. +Within one worktree, concurrent tool invocations serialize through git's own ref +locking — `update-ref` with the expected old value; on failure, re-read and +retry the append. No legitimate non-fast-forward ref update exists anywhere in the system. ### Server-side receive gate -A git server that accepts pushes holds a replica of the store, but plain -git applies no special rules to `refs/jp/issues/*`: a pusher with write -access can force-push a rewritten chain, and the server accepts what every -jp tool would refuse. -A `pre-receive` hook closes the gap by rejecting any update to -`refs/jp/issues/*` that fails to fast-forward the ref's current value, -contains a commit with more than one parent, or contains a commit whose -signature does not match the key hash in the ref path — the same checks -the fold applies at read time, run at push time. -The hook is stock git tooling: any server whose operator can install -`pre-receive` hooks can run it. +A git server that accepts pushes holds a replica of the store, but plain git +applies no special rules to `refs/jp/issues/*`: a pusher with write access can +force-push a rewritten chain, and the server accepts a rewritten chain that +every jp tool would refuse. +On servers whose operator can install hooks — self-hosted git, or GitHub +Enterprise Server — a `pre-receive` hook rejects any update to +`refs/jp/issues/*` that fails to fast-forward the ref's current value, contains +a commit with more than one parent, or contains a commit whose signature does +not match the key hash in the ref path. +The hook runs at push time the same checks the fold runs at read time. + +github.com runs no user-supplied `pre-receive` hooks, and github.com branch +protections and rulesets cover `refs/heads/*` and `refs/tags/*` only. +A store whose shared remote is github.com has no receive gate: any collaborator +with write access can force-push a rewritten chain to the shared remote. +The client-side defenses still hold: every replica's non-forcing fetch refuses +a rewritten chain, the ref journal records a rewrite forced through by hand, +and a fresh clone cannot compute issue state, because surviving chains +acknowledge ops that the rewritten chain no longer carries. +The receive gate is extra hardening on servers that support hooks; the design +does not depend on the receive gate. ### The scraper @@ -298,20 +333,20 @@ Phase 1 scrapes issues and their conversation comments (comment bodies are editable on GitHub, hence `set-comment-body`). The scraper also detects upstream deletions, under one rule: a tombstone -requires positive evidence of deletion, never absence from a -possibly-incomplete listing. -Edits are positive observations and are committed incrementally as scraped; -a deletion is an inference from absence, so deletion ops are emitted only -after the enumeration they are inferred from has run to completion, and -only after a direct GitHub API request for the missing item confirms the -deletion (HTTP 404/410). -Concretely: each run enumerates the full issue list, and the comment id set -of each changed issue; a previously observed issue or comment missing from -its completed enumeration is requested individually from the API, and only -an HTTP 404/410 yields the `delete-issue` or `delete-comment` op, carrying -scraper provenance. -A scrape that aborts mid-run (rate limit, network failure) therefore emits -the edits it observed and no deletions. +requires positive evidence of deletion, never absence from a possibly-incomplete +listing. +Edits are positive observations and are committed incrementally as scraped; a +deletion is an inference from absence, so deletion ops are emitted only after +the enumeration they are inferred from has run to completion, and only after a +direct GitHub API request for the missing item confirms the deletion (HTTP +404/410). +Concretely: each run enumerates the full issue list, and the comment id set of +each changed issue; a previously observed issue or comment missing from its +completed enumeration is requested individually from the API, and only an HTTP +404/410 yields the `delete-issue` or `delete-comment` op, carrying scraper +provenance. +A scrape that aborts mid-run (rate limit, network failure) therefore emits the +edits it observed and no deletions. ### Signed commits @@ -319,21 +354,20 @@ Every writer signs its commits (`git commit-tree -S`; SSH-key signing via `gpg.format=ssh` keeps the requirement to a key writers already have). The scraper signs like any other writer. -Enforcement happens at read time, inside the fold — commits can arrive from -any remote or bundle, so no single point exists through which all writes -pass, and GitHub's signed-commit protections only cover branches. +Enforcement happens at read time, inside the fold — commits can arrive from any +remote or bundle, so no single point exists through which all writes pass, and +GitHub's signed-commit protections only cover branches. The fold verifies each commit (`git verify-commit`); ops from unverifiable commits are excluded from the computed state and surfaced as a warning. A configuration option (on by default) escalates the warning to a hard failure. Verification is required only from a configured cutoff. -A `verification_required = ""` configuration field names a -commit: that commit and everything after it (in the causal order of the -section "Computing issue state") require verification, while commits before -the cutoff are exempt from the checks above. +A `verification_required = ""` configuration field names a commit: +that commit and everything after it (in the causal order of the section +"Computing issue state") require verification, while commits before the cutoff +are exempt from the checks above. With the field unset, no commit requires verification. -This lets a store adopt signing late without invalidating its earlier -history. +This lets a store adopt signing late without invalidating its earlier history. The trust anchor stays **outside the repository**: a per-user allowed-signers file (git's `gpg.ssh.allowedSignersFile` format) under the user's own @@ -342,28 +376,27 @@ A checked-in list would let anyone who can push code appoint signers — the artifact being verified must not control its own trust anchor. The signing key is the authoritative identity, and the writer id embeds its -hash: the fold rejects commits that live under a key-hash segment but are -not signed by the matching key. +hash: the fold rejects commits that live under a key-hash segment but are not +signed by the matching key. All worktree refs under the same key hash belong to the same principal. ### Deletion, redaction, and capabilities The store is append-only: no ref is ever rewritten and no object is ever removed. -Every replica refuses a non-fast-forward update of a foreign ref, so an -author cannot hide a history rewrite — existing replicas detect it -directly, and dangling `seen_heads` references expose it even to fresh -clones. +Every replica refuses a non-fast-forward update of a foreign ref, so an author +cannot hide a history rewrite — existing replicas detect the rewrite +directly, and on a fresh clone the fold stops on the dangling `seen_heads` +references that the rewrite leaves behind. `delete-issue`, `delete-comment`, and `redact-op` are ordinary ops: they -propagate like any other, and every fold hides the target from that point -on. +propagate like any other, and every fold hides the target from that point on. The hidden bytes remain in every clone, permanently. -**Keeping deleted content.** A user's delete is not final: any single -moderator may keep the deleted messages by recording a `keep` op in their -own chain, naming the tombstoned target and the rendering they chose — -full text, a marker, or a moderator-written summary. +**Keeping deleted content.** A user's delete is not final: any single moderator +may keep the deleted messages by recording a `keep` op in their own chain, +naming the tombstoned target and the rendering they chose — full text, a +marker, or a moderator-written summary. Nothing needs rescuing: the content is still in the author's chain, merely hidden, and the `keep` op changes how the fold renders it. @@ -372,14 +405,13 @@ time. The capability policy lives with the allowed-signers file, outside the repository, consistent with the trust anchor decision. The table below lists *defaults*: the policy format expresses other rules — -different thresholds than 1-of-n, per-action writer sets — without any -change to the store format, because capabilities are evaluated at fold time -and never recorded in ops. +different thresholds than 1-of-n, per-action writer sets — without any change +to the store format, because capabilities are evaluated at fold time and never +recorded in ops. Ops signed by a key lacking the required capability are excluded from the -computed state and surfaced, with the same warn/enforce handling as -unverifiable signatures. -The originator of an issue or comment is the key that signed its creating -op. +computed state and surfaced, with the same warn/enforce handling as unverifiable +signatures. +The originator of an issue or comment is the key that signed its creating op. | Action | Op | Default capability | | --- | --- | --- | @@ -401,51 +433,50 @@ Store-level metadata (the allowed-labels vocabulary) lives in its own log at All object and ref access shells out to the `git` binary through the existing `ProcessRunner` abstraction in the tools crate: -- writes: `git hash-object -w --stdin`, `git mktree`, `git commit-tree`, - `git update-ref` -- reads: `git for-each-ref`, `git rev-list`, one-shot - `git cat-file --batch` with all requests written to stdin upfront +- writes: `git hash-object -w --stdin`, `git mktree`, `git commit-tree`, `git + update-ref` +- reads: `git for-each-ref`, `git rev-list`, one-shot `git cat-file --batch` + with all requests written to stdin upfront Rationale, in order of weight: -1. **Coexistence is correctness-critical.** The store lives inside - repositories people care about. +1. **Coexistence is correctness-critical.** The store lives inside repositories + people care about. The git binary can never disagree with itself about locking, gc, packfile - formats, or the repo's object format (SHA-256 repos are inherited for - free). -2. **Scale does not justify a library.** Hundreds of issues, dozens of ops - each; incremental scrapes write a handful of commits. + formats, or the repo's object format (SHA-256 repos are inherited for free). +2. **Scale does not justify a library.** Hundreds of issues, dozens of ops each; + incremental scrapes write a handful of commits. The initial import is a one-time bulk write of a few thousand spawns. 3. **Zero new dependencies** in a project that runs cargo-vet, and the subprocess pattern — including `MockProcessRunner` tests and real-git integration tests — already exists in the tools crate. -This decision has a pre-agreed revision trigger: if computing state across -all issues (the kanban view) measures slow, the read path moves to the `gix` -crate (reading is its most mature half) while writes stay as plumbing. +This decision has a pre-agreed revision trigger: if computing state across all +issues (the kanban view) measures slow, the read path moves to the `gix` crate +(reading is its most mature half) while writes stay as plumbing. The on-disk format is git's either way, so stored data does not change. ## Drawbacks -- Refs grow with issues × worktrees and are permanent: a chain's ops are - part of issue state, so refs cannot be pruned. +- Refs grow with issues × worktrees and are permanent: a chain's ops are part of + issue state, so refs cannot be pruned. - The store only grows. Deleted and redacted content still occupies space in every clone forever. - Reads assemble N writer heads per issue instead of walking one DAG, and - cross-writer causality is invisible to `git log --graph` — only the fold - can reconstruct it. + cross-writer causality is invisible to `git log --graph` — only the fold can + reconstruct it. Acceptable: the consumers are exclusively our own tools. - Ops carry a small map of writer id to op id (the `seen_heads` field). -- Every writer whose commits fall under the `verification_required` cutoff - must have commit signing configured, including scraper automation. -- Subprocess-based reads put a performance ceiling on computing state over - many issues; the Implementation section names the measured trigger for - moving reads to `gix`. +- Every writer whose commits fall under the `verification_required` cutoff must + have commit signing configured, including scraper automation. +- Subprocess-based reads put a performance ceiling on computing state over many + issues; the Implementation section names the measured trigger for moving reads + to `gix`. ## Alternatives -- **Checked-in files** (e.g. `issues/*.json` in the worktree): merge - conflicts in worktrees, issue churn pollutes code history. +- **Checked-in files** (e.g. `issues/*.json` in the worktree): merge conflicts + in worktrees, issue churn pollutes code history. Rejected in Motivation. - **A dedicated bare repository** owned by JP (or a radicle-style centralized store): breaks "pull and you have everything", and centralizing many @@ -459,72 +490,69 @@ The on-disk format is git's either way, so stored data does not change. pushes interpretation into the fold and makes local edits a second, differently-shaped op family. Fine-grained ops keep `issue append` symmetrical. -- **One shared ref per issue, merge-on-push**: with N replicas syncing - pairwise at arbitrary times, shared-ref convergence mints bookkeeping merge - commits at every divergent sync, and independent joins of the same heads - themselves diverge. +- **One shared ref per issue, merge-on-push**: with N replicas syncing pairwise + at arbitrary times, shared-ref convergence mints bookkeeping merge commits at + every divergent sync, and independent joins of the same heads themselves + diverge. Per-writer refs eliminate the entire category. - **Causality as commit parents** (multi-parent commits referencing foreign heads): structurally merge commits, which this design forbids; the `seen_heads` field carries the same information while keeping every chain linear. -- **git-bug / git-appraise**: closest prior art, same refs-in-repo approach, - but git-bug's elaborated last-write-wins hides conflicts that drop data. -- **`gix` or `git2` instead of plumbing subprocesses**: see the rationale - table in Design; a large vet surface (`gix`) or a C dependency (`git2`) - buys speed the workload does not need, at coexistence risk the store cannot - afford. -- **`git fast-import` for bulk writes**: a second command language to - generate and debug; the write volume does not demand it. +- **git-bug / git-appraise**: closest prior art, same refs-in-repo approach, but + git-bug's elaborated last-write-wins hides conflicts that drop data. +- **`gix` or `git2` instead of plumbing subprocesses**: see the rationale table + in Design; a large vet surface (`gix`) or a C dependency (`git2`) buys speed + the workload does not need, at coexistence risk the store cannot afford. +- **`git fast-import` for bulk writes**: a second command language to generate + and debug; the write volume does not demand it. Reach for it only if initial import time annoys someone. ## Non-Goals - Pull requests, review comments, and reactions. - Local issue creation and editing (`issue append`). - The op vocabulary and store format are built for it, but the write path is - a later phase. + The op vocabulary and store format are built for it, but the write path is a + later phase. - The kanban tool and any state caching for it (the `set-priority` op is registered here; the tool that consumes it is not). - Cross-repo mirroring. -- Moderation governance beyond the default capability rules: vote - thresholds, disputes between moderators, appeals. - The policy format is built to express these later; this RFD fixes only - the defaults. +- Moderation governance beyond the default capability rules: vote thresholds, + disputes between moderators, appeals. + The policy format is built to express these later; this RFD fixes only the + defaults. - Physical removal of store content: deletion only hides. ## Risks -- **Deleted content persists in every clone.** Every clone permanently - holds every op ever synced, including content its author deleted and - content a moderator redacted. - This is deliberate — the store is append-only — but it means true - erasure (leaked credentials, legal demands) is impossible inside the - system. +- **Deleted content persists in every clone.** Every clone permanently holds + every op ever synced, including content its author deleted and content a + moderator redacted. + This is deliberate — the store is append-only — but it means true erasure + (leaked credentials, legal demands) is impossible inside the system. The remedy for a leaked secret is rotating the secret. - **Capability policy is per-user.** Like the allowed-signers file, the capability policy lives outside the repository, so two users can compute different folded states from the same commits. Tools must surface excluded ops, so the divergence stays visible. -- **Large payloads.** When `issue append` needs content too large for - `ops.json` (logs, screenshots), the payload goes to the `.jp/blobs/` store - of [RFD 066], referenced from the op by SHA-256. - The signed op carries the checksum, so signature verification extends to - the blob content. - Consequence to accept: blobs travel with ordinary worktree commits, not - with `issue sync` ref exchange, so an op can reference a blob its reader - has not yet pulled. +- **Large payloads.** When `issue append` needs content too large for `ops.json` + (logs, screenshots), the payload goes to the `.jp/blobs/` store of [RFD 066], + referenced from the op by SHA-256. + The signed op carries the checksum, so signature verification extends to the + blob content. + Consequence to accept: blobs travel with ordinary worktree commits, not with + `issue sync` ref exchange, so an op can reference a blob its reader has not + yet pulled. Details deferred to the append-phase RFD. -- **Withholding is undetectable.** A remote can serve truthful but stale - refs: every commit validly signed, every update a fast-forward, and the - newest ops simply absent. - A reader served only by that remote sees an issue frozen in the past, and - no mechanical check distinguishes withholding from ordinary propagation - delay. - Accepted as inherent: refs never move backward, the newer tip is adopted - as soon as any replica that has it is fetched from, and the choice of - remotes is the user's. +- **Withholding is undetectable.** A remote can serve truthful but stale refs: + every commit validly signed, every update a fast-forward, and the newest ops + simply absent. + A reader served only by that remote sees an issue frozen in the past, and no + mechanical check distinguishes withholding from ordinary propagation delay. + Accepted as inherent: refs never move backward, the newer tip is adopted as + soon as any replica that has it is fetched from, and the choice of remotes is + the user's. - **Verification cost.** `git verify-commit` per commit at read time is subprocess-heavy; verification results may need caching. Measure before optimizing. @@ -533,8 +561,8 @@ The on-disk format is git's either way, so stored data does not change. Several of the mechanisms above — the ref journal, the receive gate, the missing-acknowledgment rule — exist because of one concrete attack. -This section documents it and maps each defense to the design mechanism -that closes it. +This section documents it and maps each defense to the design mechanism that +closes it. Mallory is a developer with a git remote she controls. She rewrites the history of one writer chain under `refs/jp/issues/42/`, @@ -543,38 +571,43 @@ Alice fetches code and issue refs directly from Mallory's remote. Bob runs the git server that the team otherwise shares, and Alice has push access to it. -If Alice already holds the current value of the rewritten ref, her -non-forcing fetch refuses the update: a chain missing a commit fails to -fast-forward. +If Alice already holds the current value of the rewritten ref, her non-forcing +fetch refuses the update: a chain missing a commit fails to fast-forward. If Alice is behind, or fetching these refs for the first time, she accepts -Mallory's version — a first fetch has no prior value to compare against, -and signatures authenticate authorship of the commits that are present -without proving that the set is complete. +Mallory's version — a first fetch has no prior value to compare against, and +signatures authenticate authorship of the commits that are present without +proving that the set is complete. -Otherwise-normal git tooling exposes the attack at every point where it -would otherwise take hold or spread: +Otherwise-normal git tooling exposes the attack at every point where it would +otherwise take hold or spread: - **Fast-forward refusal** (the section "Sync"): every replica that already holds the honest ref refuses Mallory's rewrite outright. -- **The local ref journal** (the section "Sync"): if Alice bypasses the - tool with a hand-run forcing fetch, the reflog records the - non-fast-forward transition; the next `issue sync` flags it and offers to - restore the journaled prior value. -- **Missing-acknowledgment detection** (the section "Computing issue - state"): other writers' `seen_heads` still name the dropped op, so even a - fresh clone — with no prior refs to compare against — detects the - rewrite; the fold stops with a typed error and `--fix-interactive` offers - to drop the offending ref. -- **The server-side receive gate** (the section "Server-side receive - gate"): spreading the corruption through the shared server requires a - force push that Bob's `pre-receive` hook refuses. +- **The local ref journal** (the section "Sync"): if Alice bypasses the tool + with a hand-run forcing fetch, the reflog records the non-fast-forward + transition; the next `issue sync` flags it and offers to restore the journaled + prior value. +- **Missing-acknowledgment detection** (the section "Computing issue state"): + other writers' `seen_heads` still name the dropped op, so even a fresh clone + — with no prior refs to compare against — refuses to render issue state; + the fold stops with a typed error naming the missing op id. + Mallory dropped the op from every copy she controls, but the op exists on + no other remote either, so fetching from more remotes never cures the + error, and the error hardens into evidence of a rewrite. + `--fix-interactive` offers to drop the offending ref. +- **The server-side receive gate** (the section "Server-side receive gate"): + spreading the corruption through the shared server requires a force push that + Bob's `pre-receive` hook refuses. + When the team's shared server is github.com, no hook runs and the force + push succeeds; fast-forward refusal, the ref journal, and + missing-acknowledgment detection are the remaining defenses. A remote can also *withhold*: serve truthful but stale refs, every commit validly signed and every update a fast-forward, with the newest ops simply absent. That is not a rewrite and no defense above fires; it is an accepted limit, -recorded under Risks, and the sync rules in the section "Sync" guarantee -the gap heals on the next sync with any replica that has the newer state. +recorded under Risks, and the sync rules in the section "Sync" guarantee the gap +heals on the next sync with any replica that has the newer state. ## Implementation Plan @@ -583,37 +616,36 @@ Each phase is independently reviewable and mergeable. 1. **Store primitives** in the tools crate: writer-id minting and storage, `ops.json` schema (versioned), commit read/write via `ProcessRunner` plumbing, ref enumeration. - Unit-tested against `MockProcessRunner`, integration-tested against real - temp repos. + Unit-tested against `MockProcessRunner`, integration-tested against real temp + repos. 2. **State computation (the fold)**: chain walking, causal ordering from `seen_heads`, the missing-acknowledgment check with its typed error and - `--fix-interactive` resolution, deterministic tiebreak, per-field - semantics (multi-value registers, observed-remove set, grow-only set). + `--fix-interactive` resolution, deterministic tiebreak, per-field semantics + (multi-value registers, observed-remove set, grow-only set). Pure logic over data fetched by phase 1; property-style tests for order-independence. -3. **Scraper** (`issue sync`, write side): scrape via `jp_github`, diff - scraped state against computed local state, emit ops — including - `delete-issue` / `delete-comment` tombstones for upstream deletions — - and sign commits. +3. **Scraper** (`issue sync`, write side): scrape via `jp_github`, diff scraped + state against computed local state, emit ops — including `delete-issue` / + `delete-comment` tombstones for upstream deletions — and sign commits. Depends on phases 1–2. 4. **Sync** (`issue sync`, transport side): fetch refspec configuration, - fast-forward pushes, refusal to run while any remote's configured - `fetch` entry carries a forcing refspec covering `refs/jp/`, + fast-forward pushes, refusal to run while any remote's configured `fetch` + entry carries a forcing refspec covering `refs/jp/`, `core.logAllRefUpdates=always` setup and the ref-journal check, and a reference `pre-receive` hook for server operators. -5. **`issue show`**: compute and render one issue's state, including - surfaced multi-value conflicts and unverified-writer warnings. -6. **Signature verification** during state computation: `verify-commit` - against the per-user allowed-signers file, with the warn/enforce - configuration option and the `verification_required` cutoff. +5. **`issue show`**: compute and render one issue's state, including surfaced + multi-value conflicts and unverified-writer warnings. +6. **Signature verification** during state computation: `verify-commit` against + the per-user allowed-signers file, with the warn/enforce configuration option + and the `verification_required` cutoff. 7. **Deletion and keeps** (`issue delete`): tombstone ops, `keep` ops, capability checks against the per-user policy. ## References - [RFD 066] — Content-Addressable Blob Store: content-addressed storage for - conversation blobs, and the designated home for large payloads in the - later `issue append` phase (see the section "Risks"). + conversation blobs, and the designated home for large payloads in the later + `issue append` phase (see the section "Risks"). - [git-bug] — issues as git objects in refs, closest prior art. - [git-appraise] — code review as git objects in refs. From 4005bda3915ddb88e59c5ffb2d5d2570c52c6361 Mon Sep 17 00:00:00 2001 From: rgrant Date: Fri, 24 Jul 2026 20:21:44 +0000 Subject: [PATCH 05/10] 20260724 changes after reviewing Parasyte's comments --- .../D24-git-object-store-for-github-issues.md | 168 ++++++++---------- 1 file changed, 73 insertions(+), 95 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index 2382cf2e..6d4be2a6 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -22,12 +22,14 @@ to edit them locally (`issue append`) and build views over them (a kanban tool). The consumers are our own jp-tools: `issue show` first, `issue append` and a kanban view later. -GitHub is a read-only interface where some issues get added. +In this version of the design, GitHub is an ingest source only: issues filed +there are scraped into the local store, but the tooling never writes back to +GitHub. It is not the authority. The local store is the source of truth, designed from day one for a world where local edits are made concurrently on machines that do not know about each other. -Three storage approaches fail the requirements: +Three approaches fail the requirements: - **Checked-in files** cause merge conflicts for people working in worktrees, and dirty the code's evolution with their own commit history. @@ -81,18 +83,24 @@ refs/jp/issues// The literal `meta` is reserved for the store-level metadata log. The store holds the repo's own issues only; a cross-repo mirror would get a sibling namespace and is out of scope. -- `` identifies one worktree of one clone, held by one signer. - It is composed of three segments, - `/-/`: - the workspace id scopes the writer to the JP workspace that produced it; the - key hash binds the ref to a signing key (the fold rejects commits under a +- `` identifies one replica: one checkout of the repository — a + clone, or a linked git worktree inside a clone — held by one signer. + It is composed of two segments, + `-/`: + the key hash binds the ref to a signing key (the fold rejects commits under a key-hash segment that are not signed by the matching key); the avatar nickname - is a human-readable label with no authority; the worktree id is a - `jp_id`-formatted id minted per worktree on first write, stored in that - worktree's own state under `.git/`, never checked in and never synced. - Every worktree of every clone is its own writer. - -A ref is only ever written by its owner — one worktree — so no two replicas + is a human-readable label with no authority; the replica id is a + `jp_id`-formatted id minted per checkout on first write, stored in that + checkout's own state under `.git/`, never checked in and never synced. + The replica id exists because one signer works from several checkouts at + once: if the signing key alone named the writer, two checkouts holding the + same key would append to the same ref independently, and one of the two + pushes would fail to fast-forward. + A replica id per checkout gives each checkout its own chain, so every push + stays a fast-forward. + Every checkout of every clone is its own writer. + +A ref is only ever written by its owner — one replica — so no two replicas ever contend on a ref. Every push is a fast-forward, and no merge commit exists anywhere in the store. @@ -132,7 +140,7 @@ issue state". Operations are fine-grained, one vocabulary for scraped and (later) local writes. Each op carries a stable `jp_id`-formatted `id`, minted at creation; the id is -what tombstones and redaction stubs reference. +what `seen_heads` acknowledgments reference. | Op | Target | Fold semantics | | --- | --- | --- | @@ -144,15 +152,13 @@ what tombstones and redaction stubs reference. | `set-comment-body` | comment (by GitHub id) | multi-value register | | `set-comment-visibility` | comment (by GitHub id) | multi-value register | | `set-priority` | issue | multi-value register | -| `delete-issue` | issue | tombstone | -| `delete-comment` | comment (by GitHub id) | tombstone | -| `redact-op` | op (by op id) | tombstone | -| `keep` | issue or comment | keep (defined below) | +| `tombstone-issue` | issue | tombstone | +| `tombstone-comment` | comment (by GitHub id) | tombstone | | `set-allowed-labels` | store metadata | multi-value register | A register is one field of one target: ops of the same type on the same target address the same register. -The three fold semantics named in the table are defined in the section +The fold semantics named in the table are defined in the section "Computing issue state", after the causal order they depend on. `set-comment-visibility` records comment collapsing — GitHub's @@ -191,19 +197,22 @@ This relation is the only input to conflict resolution. The register holds a single value again only when a later write acknowledges every op in that set — that write happens-after each of them, and the fold resolves the register to it. + A resolving write is an ordinary write, so two concurrent resolving writes + produce a new conflict containing only the resolving values; the same rule + applies until one write acknowledges every live value. - **Observed-remove set** (labels): a label is present when some `add-label` op for it does not happen-before a `remove-label` op for it. A remove affects only adds it acknowledged; a concurrent add survives. + A removed label can be re-added by a later `add-label` op; removal is never + permanent (the distinction from a two-phase set, where removal is final). - **Grow-only set** (comments): the union of `add-comment` ops. -- **Tombstone** (deletes, redactions): once present, every fold hides the target +- **Tombstone** (deletes): once present, every fold hides the target from that point on; concurrent edits to the target stay in the log but are not displayed. Hiding is the entire mechanism; the bytes are never removed from the store. -- **Keep** (moderation): a `keep` op from any single moderator (1-of-n) - overrides a tombstone on the same target. - The kept content renders however the keeping moderator chose: full text, a - marker, or a moderator-written summary. - Without a `keep`, a single redact or delete suffices to hide (also 1-of-n). + A tombstone is final: no op un-hides a tombstoned target. + When a tombstone turns out to be wrong, the original author resubmits the + content as a new issue or comment. **Missing acknowledgments.** Before applying the resolution rules, the fold checks every `seen_heads` reference. @@ -289,9 +298,9 @@ Syncing with a participant who is behind carries no penalty, and which remotes to fetch from stays the user's decision. Two replicas may push the same ref, but a push race cannot corrupt a chain: only -the owning worktree ever appends commits to a chain, so competing pushes carry +the owning replica ever appends commits to a chain, so competing pushes carry the same tip, or one tip is an extension of the other. -Within one worktree, concurrent tool invocations serialize through git's own ref +Within one replica, concurrent tool invocations serialize through git's own ref locking — `update-ref` with the expected old value; on failure, re-read and retry the append. No legitimate non-fast-forward ref update exists anywhere in the system. @@ -313,10 +322,10 @@ github.com runs no user-supplied `pre-receive` hooks, and github.com branch protections and rulesets cover `refs/heads/*` and `refs/tags/*` only. A store whose shared remote is github.com has no receive gate: any collaborator with write access can force-push a rewritten chain to the shared remote. -The client-side defenses still hold: every replica's non-forcing fetch refuses -a rewritten chain, the ref journal records a rewrite forced through by hand, -and a fresh clone cannot compute issue state, because surviving chains -acknowledge ops that the rewritten chain no longer carries. +The client-side defenses still hold: every replica's non-forcing fetch refuses a +rewritten chain, the ref journal records a rewrite forced through by hand, and a +fresh clone cannot compute issue state, because surviving chains acknowledge ops +that the rewritten chain no longer carries. The receive gate is extra hardening on servers that support hooks; the design does not depend on the receive gate. @@ -343,8 +352,8 @@ direct GitHub API request for the missing item confirms the deletion (HTTP Concretely: each run enumerates the full issue list, and the comment id set of each changed issue; a previously observed issue or comment missing from its completed enumeration is requested individually from the API, and only an HTTP -404/410 yields the `delete-issue` or `delete-comment` op, carrying scraper -provenance. +404/410 yields the `tombstone-issue` or `tombstone-comment` op, carrying +scraper provenance. A scrape that aborts mid-run (rate limit, network failure) therefore emits the edits it observed and no deletions. @@ -378,52 +387,28 @@ artifact being verified must not control its own trust anchor. The signing key is the authoritative identity, and the writer id embeds its hash: the fold rejects commits that live under a key-hash segment but are not signed by the matching key. -All worktree refs under the same key hash belong to the same principal. +All replica refs under the same key hash belong to the same principal. -### Deletion, redaction, and capabilities +### Deletion The store is append-only: no ref is ever rewritten and no object is ever removed. Every replica refuses a non-fast-forward update of a foreign ref, so an author -cannot hide a history rewrite — existing replicas detect the rewrite -directly, and on a fresh clone the fold stops on the dangling `seen_heads` -references that the rewrite leaves behind. +cannot hide a history rewrite — existing replicas detect the rewrite directly, +and on a fresh clone the fold stops on the dangling `seen_heads` references that +the rewrite leaves behind. -`delete-issue`, `delete-comment`, and `redact-op` are ordinary ops: they -propagate like any other, and every fold hides the target from that point on. +`tombstone-issue` and `tombstone-comment` are ordinary ops: they propagate +like any other, and every fold hides the target from that point on. The hidden bytes remain in every clone, permanently. -**Keeping deleted content.** A user's delete is not final: any single moderator -may keep the deleted messages by recording a `keep` op in their own chain, -naming the tombstoned target and the rendering they chose — full text, a -marker, or a moderator-written summary. -Nothing needs rescuing: the content is still in the author's chain, merely -hidden, and the `keep` op changes how the fold renders it. - -**Capabilities.** Actions are authorized per signing key, evaluated at fold -time. -The capability policy lives with the allowed-signers file, outside the -repository, consistent with the trust anchor decision. -The table below lists *defaults*: the policy format expresses other rules — -different thresholds than 1-of-n, per-action writer sets — without any change -to the store format, because capabilities are evaluated at fold time and never -recorded in ops. -Ops signed by a key lacking the required capability are excluded from the -computed state and surfaced, with the same warn/enforce handling as unverifiable -signatures. -The originator of an issue or comment is the key that signed its creating op. - -| Action | Op | Default capability | -| --- | --- | --- | -| close / reopen issue | `set-state` | any trusted writer | -| tag / untag (incl. `wontfix`) | `add-label` / `remove-label` | any trusted writer | -| reprioritize (kanban ordering) | `set-priority` | any trusted writer | -| edit allowed tags | `set-allowed-labels` | moderators | -| collapse / un-collapse comment | `set-comment-visibility` | moderators | -| moderate issue / comment | `redact-op` | any single moderator (1-of-n) | -| keep deleted content | `keep` | any single moderator (1-of-n) | -| delete issue | `delete-issue` | originator while no other writer has appended activity; moderators otherwise | -| delete comment | `delete-comment` | originator, subject to moderator keep; moderators | +Every writer trusted by the allowed-signers file is a full peer: any peer +may perform any op on any item, including a tombstone on an item another +peer created. +A tombstone is final; no op un-hides a tombstoned target. +When a tombstone turns out to be wrong, the original author resubmits the +content as a new issue or comment, and the new item can be tombstoned in +turn by the same rule. Store-level metadata (the allowed-labels vocabulary) lives in its own log at `refs/jp/issues/meta/`, using the same op machinery as an issue. @@ -458,10 +443,10 @@ The on-disk format is git's either way, so stored data does not change. ## Drawbacks -- Refs grow with issues × worktrees and are permanent: a chain's ops are part of +- Refs grow with issues × replicas and are permanent: a chain's ops are part of issue state, so refs cannot be pruned. - The store only grows. - Deleted and redacted content still occupies space in every clone forever. + Deleted content still occupies space in every clone forever. - Reads assemble N writer heads per issue instead of walking one DAG, and cross-writer causality is invisible to `git log --graph` — only the fold can reconstruct it. @@ -517,24 +502,18 @@ The on-disk format is git's either way, so stored data does not change. - The kanban tool and any state caching for it (the `set-priority` op is registered here; the tool that consumes it is not). - Cross-repo mirroring. -- Moderation governance beyond the default capability rules: vote thresholds, - disputes between moderators, appeals. - The policy format is built to express these later; this RFD fixes only the - defaults. +- Moderation and per-key authorization: every trusted writer is a full peer. + Authorization rules can be added later, evaluated at fold time, without + changing stored data. - Physical removal of store content: deletion only hides. ## Risks - **Deleted content persists in every clone.** Every clone permanently holds - every op ever synced, including content its author deleted and content a - moderator redacted. + every op ever synced, including content its author deleted. This is deliberate — the store is append-only — but it means true erasure (leaked credentials, legal demands) is impossible inside the system. The remedy for a leaked secret is rotating the secret. -- **Capability policy is per-user.** Like the allowed-signers file, the - capability policy lives outside the repository, so two users can compute - different folded states from the same commits. - Tools must surface excluded ops, so the divergence stays visible. - **Large payloads.** When `issue append` needs content too large for `ops.json` (logs, screenshots), the payload goes to the `.jp/blobs/` store of [RFD 066], referenced from the op by SHA-256. @@ -589,18 +568,18 @@ otherwise take hold or spread: prior value. - **Missing-acknowledgment detection** (the section "Computing issue state"): other writers' `seen_heads` still name the dropped op, so even a fresh clone - — with no prior refs to compare against — refuses to render issue state; - the fold stops with a typed error naming the missing op id. - Mallory dropped the op from every copy she controls, but the op exists on - no other remote either, so fetching from more remotes never cures the - error, and the error hardens into evidence of a rewrite. + — with no prior refs to compare against — refuses to render issue state; the + fold stops with a typed error naming the missing op id. + Mallory dropped the op from every copy she controls, but the op exists on no + other remote either, so fetching from more remotes never cures the error, and + the error hardens into evidence of a rewrite. `--fix-interactive` offers to drop the offending ref. - **The server-side receive gate** (the section "Server-side receive gate"): spreading the corruption through the shared server requires a force push that Bob's `pre-receive` hook refuses. - When the team's shared server is github.com, no hook runs and the force - push succeeds; fast-forward refusal, the ref journal, and - missing-acknowledgment detection are the remaining defenses. + When the team's shared server is github.com, no hook runs and the force push + succeeds; fast-forward refusal, the ref journal, and missing-acknowledgment + detection are the remaining defenses. A remote can also *withhold*: serve truthful but stale refs, every commit validly signed and every update a fast-forward, with the newest ops simply @@ -625,8 +604,8 @@ Each phase is independently reviewable and mergeable. Pure logic over data fetched by phase 1; property-style tests for order-independence. 3. **Scraper** (`issue sync`, write side): scrape via `jp_github`, diff scraped - state against computed local state, emit ops — including `delete-issue` / - `delete-comment` tombstones for upstream deletions — and sign commits. + state against computed local state, emit ops — including `tombstone-issue` + / `tombstone-comment` for upstream deletions — and sign commits. Depends on phases 1–2. 4. **Sync** (`issue sync`, transport side): fetch refspec configuration, fast-forward pushes, refusal to run while any remote's configured `fetch` @@ -638,8 +617,7 @@ Each phase is independently reviewable and mergeable. 6. **Signature verification** during state computation: `verify-commit` against the per-user allowed-signers file, with the warn/enforce configuration option and the `verification_required` cutoff. -7. **Deletion and keeps** (`issue delete`): tombstone ops, `keep` ops, - capability checks against the per-user policy. +7. **Deletion** (`issue delete`): tombstone ops. ## References From 422d216c1b9b5e05161c1696c6fb84fc5eb07408 Mon Sep 17 00:00:00 2001 From: rgrant Date: Fri, 24 Jul 2026 21:09:17 +0000 Subject: [PATCH 06/10] 20260724 clarify tool invocation and ref namespaces --- .../D24-git-object-store-for-github-issues.md | 48 ++++++++++++++----- 1 file changed, 35 insertions(+), 13 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index 6d4be2a6..b96d672b 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -19,7 +19,7 @@ dropped data. Contributors need offline access to the project's issues, and later the ability to edit them locally (`issue append`) and build views over them (a kanban tool). -The consumers are our own jp-tools: `issue show` first, `issue append` and a +The consumers are our own tooling: `jp issue show` first, `issue append` and a kanban view later. In this version of the design, GitHub is an ingest source only: issues filed @@ -52,10 +52,10 @@ need for offline work. ```sh # sync: fetch everyone's issue refs, push every issue ref you hold -jp-tools issue sync +jp issue sync # read a folded issue, offline -jp-tools issue show 42 +jp issue show 42 # poke at the raw store with stock git git for-each-ref refs/jp/issues/42/ @@ -100,6 +100,12 @@ refs/jp/issues// stays a fast-forward. Every checkout of every clone is its own writer. +`refs/jp/issues/` is an ordinary ref hierarchy in the repository's normal ref +space, like git's own `refs/notes/*` — not the ref-remapping mechanism of +[gitnamespaces(1)], which serves multiple logical repositories from one object +store by rewriting every ref name server-side. +Nothing here is remapped: clients see, fetch, and push the literal ref names. + A ref is only ever written by its owner — one replica — so no two replicas ever contend on a ref. Every push is a fast-forward, and no merge commit exists anywhere in the store. @@ -413,10 +419,20 @@ turn by the same rule. Store-level metadata (the allowed-labels vocabulary) lives in its own log at `refs/jp/issues/meta/`, using the same op machinery as an issue. +### Where the code lives + +The tooling is an internal command plugin: a `jp-issue` binary under +`crates/internal/`, registering the `issue` command path, so the commands above +run as `jp issue sync` and `jp issue show`. +The plugin is built and installed locally via the existing plugin +infrastructure and is not published to the plugin registry. +Store primitives, the fold, and the scraper are library code inside the same +crate; the CLI is a thin shell over that library, and the fold stays pure logic +over data the store layer fetched. + ### Implementation: git plumbing subprocesses -All object and ref access shells out to the `git` binary through the existing -`ProcessRunner` abstraction in the tools crate: +All object and ref access shells out to the `git` binary: - writes: `git hash-object -w --stdin`, `git mktree`, `git commit-tree`, `git update-ref` @@ -432,9 +448,7 @@ Rationale, in order of weight: 2. **Scale does not justify a library.** Hundreds of issues, dozens of ops each; incremental scrapes write a handful of commits. The initial import is a one-time bulk write of a few thousand spawns. -3. **Zero new dependencies** in a project that runs cargo-vet, and the - subprocess pattern — including `MockProcessRunner` tests and real-git - integration tests — already exists in the tools crate. +3. **Zero new dependencies** in a project that runs cargo-vet. This decision has a pre-agreed revision trigger: if computing state across all issues (the kanban view) measures slow, the read path moves to the `gix` crate @@ -507,6 +521,12 @@ The on-disk format is git's either way, so stored data does not change. changing stored data. - Physical removal of store content: deletion only hides. +## Future Work + +- Assistant-facing `issue_*` tools: thin, non-interactive read tools (an + `issue_show`, for example) layered on the store library, so the assistant can + consult the issue store during conversations. + ## Risks - **Deleted content persists in every clone.** Every clone permanently holds @@ -592,11 +612,12 @@ heals on the next sync with any replica that has the newer state. Each phase is independently reviewable and mergeable. -1. **Store primitives** in the tools crate: writer-id minting and storage, - `ops.json` schema (versioned), commit read/write via `ProcessRunner` - plumbing, ref enumeration. - Unit-tested against `MockProcessRunner`, integration-tested against real temp - repos. +1. **Store primitives** in the `jp-issue` crate: the plugin skeleton and + `issue` command registration, writer-id minting and storage, `ops.json` + schema (versioned), commit read/write via git plumbing subprocesses, ref + enumeration. + Unit-tested against mocked git invocations, integration-tested against real + temporary repositories. 2. **State computation (the fold)**: chain walking, causal ordering from `seen_heads`, the missing-acknowledgment check with its typed error and `--fix-interactive` resolution, deterministic tiebreak, per-field semantics @@ -633,5 +654,6 @@ Each phase is independently reviewable and mergeable. [RFD 066]: ../066-content-addressable-blob-store.md [git-bug]: https://github.com/git-bug/git-bug +[gitnamespaces(1)]: https://git-scm.com/docs/gitnamespaces [git-appraise]: https://github.com/google/git-appraise [radicle COBs]: https://radicle.xyz/guides/protocol#collaborative-objects From 40c8e90616e12d692e5eeb1d26d1c7e593c79bd1 Mon Sep 17 00:00:00 2001 From: rgrant Date: Fri, 24 Jul 2026 21:22:39 +0000 Subject: [PATCH 07/10] 20260724 drop mirrors --- docs/rfd/drafts/D24-git-object-store-for-github-issues.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index b96d672b..9000c219 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -9,8 +9,8 @@ This RFD describes a local-first store for GitHub issues, kept as git objects in the project's own `.git` under a dedicated ref namespace. -A scraper tool mirrors issues and their comments from GitHub through -`jp_github`; contributors sync the store with ordinary push and pull. +A scraper tool ingests issues and their comments from GitHub through +`jp_github`, one way; contributors sync the store with ordinary push and pull. Each issue is a set of per-writer, append-only operation logs; issue state is computed deterministically from their union, without merge conflicts and without dropped data. From b051ba6ef6b10d73dc76502f0865e85dad569dcf Mon Sep 17 00:00:00 2001 From: rgrant Date: Fri, 24 Jul 2026 21:23:16 +0000 Subject: [PATCH 08/10] 20260724 run comfort --- .../D24-git-object-store-for-github-issues.md | 60 +++++++++---------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index 9000c219..a63a1344 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -86,16 +86,16 @@ refs/jp/issues// - `` identifies one replica: one checkout of the repository — a clone, or a linked git worktree inside a clone — held by one signer. It is composed of two segments, - `-/`: - the key hash binds the ref to a signing key (the fold rejects commits under a - key-hash segment that are not signed by the matching key); the avatar nickname - is a human-readable label with no authority; the replica id is a - `jp_id`-formatted id minted per checkout on first write, stored in that - checkout's own state under `.git/`, never checked in and never synced. - The replica id exists because one signer works from several checkouts at - once: if the signing key alone named the writer, two checkouts holding the - same key would append to the same ref independently, and one of the two - pushes would fail to fast-forward. + `-/`: the key hash + binds the ref to a signing key (the fold rejects commits under a key-hash + segment that are not signed by the matching key); the avatar nickname is a + human-readable label with no authority; the replica id is a `jp_id`-formatted + id minted per checkout on first write, stored in that checkout's own state + under `.git/`, never checked in and never synced. + The replica id exists because one signer works from several checkouts at once: + if the signing key alone named the writer, two checkouts holding the same key + would append to the same ref independently, and one of the two pushes would + fail to fast-forward. A replica id per checkout gives each checkout its own chain, so every push stays a fast-forward. Every checkout of every clone is its own writer. @@ -164,8 +164,8 @@ what `seen_heads` acknowledgments reference. A register is one field of one target: ops of the same type on the same target address the same register. -The fold semantics named in the table are defined in the section -"Computing issue state", after the causal order they depend on. +The fold semantics named in the table are defined in the section "Computing +issue state", after the causal order they depend on. `set-comment-visibility` records comment collapsing — GitHub's comment-minimization feature — as a register holding `visible` or `collapsed` @@ -212,8 +212,8 @@ This relation is the only input to conflict resolution. A removed label can be re-added by a later `add-label` op; removal is never permanent (the distinction from a two-phase set, where removal is final). - **Grow-only set** (comments): the union of `add-comment` ops. -- **Tombstone** (deletes): once present, every fold hides the target - from that point on; concurrent edits to the target stay in the log but are not +- **Tombstone** (deletes): once present, every fold hides the target from that + point on; concurrent edits to the target stay in the log but are not displayed. Hiding is the entire mechanism; the bytes are never removed from the store. A tombstone is final: no op un-hides a tombstoned target. @@ -358,8 +358,8 @@ direct GitHub API request for the missing item confirms the deletion (HTTP Concretely: each run enumerates the full issue list, and the comment id set of each changed issue; a previously observed issue or comment missing from its completed enumeration is requested individually from the API, and only an HTTP -404/410 yields the `tombstone-issue` or `tombstone-comment` op, carrying -scraper provenance. +404/410 yields the `tombstone-issue` or `tombstone-comment` op, carrying scraper +provenance. A scrape that aborts mid-run (rate limit, network failure) therefore emits the edits it observed and no deletions. @@ -404,17 +404,17 @@ cannot hide a history rewrite — existing replicas detect the rewrite directly, and on a fresh clone the fold stops on the dangling `seen_heads` references that the rewrite leaves behind. -`tombstone-issue` and `tombstone-comment` are ordinary ops: they propagate -like any other, and every fold hides the target from that point on. +`tombstone-issue` and `tombstone-comment` are ordinary ops: they propagate like +any other, and every fold hides the target from that point on. The hidden bytes remain in every clone, permanently. -Every writer trusted by the allowed-signers file is a full peer: any peer -may perform any op on any item, including a tombstone on an item another -peer created. +Every writer trusted by the allowed-signers file is a full peer: any peer may +perform any op on any item, including a tombstone on an item another peer +created. A tombstone is final; no op un-hides a tombstoned target. When a tombstone turns out to be wrong, the original author resubmits the -content as a new issue or comment, and the new item can be tombstoned in -turn by the same rule. +content as a new issue or comment, and the new item can be tombstoned in turn by +the same rule. Store-level metadata (the allowed-labels vocabulary) lives in its own log at `refs/jp/issues/meta/`, using the same op machinery as an issue. @@ -424,8 +424,8 @@ Store-level metadata (the allowed-labels vocabulary) lives in its own log at The tooling is an internal command plugin: a `jp-issue` binary under `crates/internal/`, registering the `issue` command path, so the commands above run as `jp issue sync` and `jp issue show`. -The plugin is built and installed locally via the existing plugin -infrastructure and is not published to the plugin registry. +The plugin is built and installed locally via the existing plugin infrastructure +and is not published to the plugin registry. Store primitives, the fold, and the scraper are library code inside the same crate; the CLI is a thin shell over that library, and the fold stays pure logic over data the store layer fetched. @@ -612,9 +612,9 @@ heals on the next sync with any replica that has the newer state. Each phase is independently reviewable and mergeable. -1. **Store primitives** in the `jp-issue` crate: the plugin skeleton and - `issue` command registration, writer-id minting and storage, `ops.json` - schema (versioned), commit read/write via git plumbing subprocesses, ref +1. **Store primitives** in the `jp-issue` crate: the plugin skeleton and `issue` + command registration, writer-id minting and storage, `ops.json` schema + (versioned), commit read/write via git plumbing subprocesses, ref enumeration. Unit-tested against mocked git invocations, integration-tested against real temporary repositories. @@ -625,8 +625,8 @@ Each phase is independently reviewable and mergeable. Pure logic over data fetched by phase 1; property-style tests for order-independence. 3. **Scraper** (`issue sync`, write side): scrape via `jp_github`, diff scraped - state against computed local state, emit ops — including `tombstone-issue` - / `tombstone-comment` for upstream deletions — and sign commits. + state against computed local state, emit ops — including `tombstone-issue` / + `tombstone-comment` for upstream deletions — and sign commits. Depends on phases 1–2. 4. **Sync** (`issue sync`, transport side): fetch refspec configuration, fast-forward pushes, refusal to run while any remote's configured `fetch` From a41f0f59cff69bd7c6db5f60d51bb1ba489f65ad Mon Sep 17 00:00:00 2001 From: rgrant Date: Fri, 24 Jul 2026 21:28:07 +0000 Subject: [PATCH 09/10] 20260724 run comfort using secret recipe --- .../D24-git-object-store-for-github-issues.md | 32 +++++++++++-------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index a63a1344..f0348d9a 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -148,19 +148,19 @@ writes. Each op carries a stable `jp_id`-formatted `id`, minted at creation; the id is what `seen_heads` acknowledgments reference. -| Op | Target | Fold semantics | -| --- | --- | --- | -| `set-title` | issue | multi-value register | -| `set-body` | issue | multi-value register | -| `set-state` | issue | multi-value register | -| `add-label` / `remove-label` | issue | observed-remove set | -| `add-comment` | comment (by GitHub id) | grow-only set | -| `set-comment-body` | comment (by GitHub id) | multi-value register | -| `set-comment-visibility` | comment (by GitHub id) | multi-value register | -| `set-priority` | issue | multi-value register | -| `tombstone-issue` | issue | tombstone | -| `tombstone-comment` | comment (by GitHub id) | tombstone | -| `set-allowed-labels` | store metadata | multi-value register | +| Op | Target | Fold semantics | +| ---------------------------- | ---------------------- | -------------------- | +| `set-title` | issue | multi-value register | +| `set-body` | issue | multi-value register | +| `set-state` | issue | multi-value register | +| `add-label` / `remove-label` | issue | observed-remove set | +| `add-comment` | comment (by GitHub id) | grow-only set | +| `set-comment-body` | comment (by GitHub id) | multi-value register | +| `set-comment-visibility` | comment (by GitHub id) | multi-value register | +| `set-priority` | issue | multi-value register | +| `tombstone-issue` | issue | tombstone | +| `tombstone-comment` | comment (by GitHub id) | tombstone | +| `set-allowed-labels` | store metadata | multi-value register | A register is one field of one target: ops of the same type on the same target address the same register. @@ -534,6 +534,7 @@ The on-disk format is git's either way, so stored data does not change. This is deliberate — the store is append-only — but it means true erasure (leaked credentials, legal demands) is impossible inside the system. The remedy for a leaked secret is rotating the secret. + - **Large payloads.** When `issue append` needs content too large for `ops.json` (logs, screenshots), the payload goes to the `.jp/blobs/` store of [RFD 066], referenced from the op by SHA-256. @@ -552,6 +553,7 @@ The on-disk format is git's either way, so stored data does not change. Accepted as inherent: refs never move backward, the newer tip is adopted as soon as any replica that has it is fetched from, and the choice of remotes is the user's. + - **Verification cost.** `git verify-commit` per commit at read time is subprocess-heavy; verification results may need caching. Measure before optimizing. @@ -647,13 +649,15 @@ Each phase is independently reviewable and mergeable. `issue append` phase (see the section "Risks"). - [git-bug] — issues as git objects in refs, closest prior art. + - [git-appraise] — code review as git objects in refs. + - [radicle COBs] — collaborative objects as commit DAGs; this design borrows the op-log idea but rejects centralized per-user storage and multi-parent causality. [RFD 066]: ../066-content-addressable-blob-store.md +[git-appraise]: https://github.com/google/git-appraise [git-bug]: https://github.com/git-bug/git-bug [gitnamespaces(1)]: https://git-scm.com/docs/gitnamespaces -[git-appraise]: https://github.com/google/git-appraise [radicle COBs]: https://radicle.xyz/guides/protocol#collaborative-objects From 03aff0fba569bd0dbeefc4ffc16550f2400ede9c Mon Sep 17 00:00:00 2001 From: rgrant Date: Sat, 25 Jul 2026 15:59:09 +0000 Subject: [PATCH 10/10] 20260725 clarify git-bug and git-appraise differences. --- .../D24-git-object-store-for-github-issues.md | 30 +++++++++++++++---- 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md index f0348d9a..9af26485 100644 --- a/docs/rfd/drafts/D24-git-object-store-for-github-issues.md +++ b/docs/rfd/drafts/D24-git-object-store-for-github-issues.md @@ -38,7 +38,8 @@ Three approaches fail the requirements: `/storage/` does) is rejected for security reasons - it would require inventing a DSL for access control. - **Last-write-wins merging** (git-bug's approach, in elaborated form) hides - conflicts by silently dropping the losing write. + conflicts: concurrent writes are totally ordered, only the winner is rendered, + and neither author is told a conflict happened. Storing issues as git objects in the repo's own `.git` — objects in the object database, refs under a dedicated namespace, nothing in the worktree — avoids @@ -495,11 +496,28 @@ The on-disk format is git's either way, so stored data does not change. diverge. Per-writer refs eliminate the entire category. - **Causality as commit parents** (multi-parent commits referencing foreign - heads): structurally merge commits, which this design forbids; the - `seen_heads` field carries the same information while keeping every chain - linear. -- **git-bug / git-appraise**: closest prior art, same refs-in-repo approach, but - git-bug's elaborated last-write-wins hides conflicts that drop data. + heads): a commit with more than one parent is a merge commit, which this + design forbids; the `seen_heads` field carries the same information while + keeping every chain linear. +- **git-bug**: closest prior art, same refs-in-repo approach — git-bug entities + sync over plain git push and pull of refs, as this design does. git-bug tracks + causality much as this design does, so both detect concurrent edits. git-bug + then tie-breaks concurrent edits by lexicographic order and compiles state as + if no conflict existed: for a register field (title, state), one edit wins the + coin-flip, the other survives in history but disappears from view, and no + interface tells either author. +- **git-appraise**: same refs-in-repo storage and push/pull sync, applied to + code review rather than issues. + Review data lives in git-notes under `refs/notes/devtools/*`, one JSON item + per line on refs that every writer shares; divergent notes are merged with + git's `cat_sort_uniq` union strategy. + Comments union cleanly (a grow-only set, as here); review requests do not: + git-appraise sorts the requests on a commit by wall-clock timestamp and takes + the newest as current. + A skewed clock reorders history, a concurrent request silently loses, and + nothing records causality. + Shared refs also require note merges; contrast with this design which forbids + merge commits by giving each writer its own ref. - **`gix` or `git2` instead of plumbing subprocesses**: see the rationale table in Design; a large vet surface (`gix`) or a C dependency (`git2`) buys speed the workload does not need, at coexistence risk the store cannot afford.