Indexed directory tree: cap lifted, per-file receipts, files:all gate - #3
Merged
Conversation
New sans-I/O crate — the deterministic foundation for tree-scale minting and
search. Three structures, each a pure function of bytes so an index blob is
content-addressable and byte-stable (I-2), and so the wasm edge can build/query
them too:
bloom.rs fixed-size (256 B), union-composable Bloom over trigrams — the prune
gate that makes deep-tree search sublinear. Parent = OR of children,
so nesting is free. Pinned seed + k, no false negatives, saturates
gracefully. Inlinable in a manifest → a prune is a pure manifest read.
trigram.rs case-folded 3-byte shingles + an inverted TrigramIndex (posting-list
intersection). "grep N files" -> "look up a few short lists and
confirm survivors." Hex-keyed wire form (folded bytes need not be
UTF-8).
dirindex.rs the Merkle directory node: files pinned by sha256, subdirs addressed
by subtree token, with recursive files/bytes totals so a node knows
its weight without a walk.
search.rs the pure decisions: prune (Bloom gate), candidates, and rank
(matches desc, then depth, then path — deterministic).
Traversal, the filesystem walk, and the blob store stay in waggle-mcp; this crate
holds only what must be deterministic and unit-testable in isolation. 22 tests,
clippy clean, largest file 299 lines.
Additive, signed-core field `tree: Option<TreeNode>` (skip-if-none, so non-tree manifests keep their exact bytes — same pattern as `extraction`). A TreeNode carries the node's content-addressed directory index, an optional trigram-index pointer, an inlined hex Bloom summary of the subtree, and recursive files/bytes totals. The bloom stays a hex string here rather than a typed `waggle-tree::Bloom` so waggle-core keeps its dependency-free leaf position; waggle-mcp parses it. MintSpec gains a `.tree(TreeNode)` builder.
…is A+B)
mint --tree now builds a Merkle hierarchy of directory NODES instead of a flat list
of per-file tokens. Two phases (tree_mint.rs):
build_node (bottom-up): pin every file's bytes eagerly (content-addressed, so a
deleted file still reads — C-1 holds), build each directory's DirIndex, trigram
index, and Bloom summary (union of children's + local files), pre-generate the
node token. Enforces a byte budget (max-bytes, default 256 MB) so nobody pins a
giant media folder by accident.
mint_nodes (top-down): mint one signed manifest per directory, parent before
child so the parent exists when a child links to it.
A file is an index entry, not a token — so the old 200-FILE cap becomes a bound on
directory nodes (few). A 1,527-file corpus mints in one call. A per-file token is
minted only lazily, when a consumer needs a standalone reference (follow-up).
mint routes tree builds early (no orphan root token). The old flat mint_tree /
mint_children / collect_files / DENIED_DIRS are deleted. read/search/coverage still
read the old shape and are rewired next.
tree_mint.rs 406 lines; clippy clean.
…ll falls
read, search, and coverage now operate on the Merkle/trigram/Bloom index instead
of per-file child tokens (tree_read.rs):
read (projection) the directory's table of contents from its DirIndex — local
files (name/size/type) and subdirs (name/token/totals). No
descent; totals come from the index.
read --file <name> one file's bytes, fetched from the content-addressed blob by
its hash, and stamped as a real read.
search ONE call spans the whole lineage: prune each subtree on its
Bloom (literal patterns), narrow to candidate files with the
node's trigram index, confirm with a real regex match, and
RANK (matches desc, then depth, then path). Each match carries
its path + owning token to drill in. Fixes "search each
subtree token separately".
coverage node-granular over the tree: file-bearing nodes read / total,
with the true file count from the root index. Pure-container
nodes are excluded (nothing of their own to read).
The old tree.rs (read_tree/lens/tree_files) and search_tree are deleted. Four
integration tests rewritten to the indexed model; the gap-fix deny-list test too.
Verified on ~/tulving/rote/docs (1,527 files, deep hierarchy): mints in one call
in 1.6s (old cap was 200); a search for "fingerprint" spans the tree and ranks
identity-binding.md (x418) first. Correct against a disk grep.
Known, documented in the design log: the 256-byte Bloom saturates for very large
subtrees, so pruning bites mainly at small/leaf nodes (correct — no false
negatives — just less pruning at extreme density); and files:all per-file coverage
awaits a served-set the payload-free log can't hold in an 8-bit mask. Both are
follow-ups, not correctness gaps.
All workspace tests pass; clippy clean; every file <= 750 lines.
…payload
Coverage on an indexed tree was node-granular ("the folder was touched"). Too
coarse for the product's own promise — "see exactly what your AI read" should mean
WHICH FILES. This makes it per-file without breaking I-1.
The unlock: I-1 forbids bytes in the log, not positions into a signed manifest.
The log already carries two such integers, both I-1-safe by their own docstrings —
`variant` (which manifest variant served) and `regions` (which contract line-ranges
a read touched). A tree node's DirIndex is a sorted, signed, immutable file list,
so "file #k of this node was read" is the same species of object: meaningless
without the signed manifest, therefore leaks nothing. The only blocker was the
8-bit width of `regions` — a width limit, not an I-1 limit.
core Event.entry: Option<u32> — the ordinal, in the owning node's signed
DirIndex file order, of the one file a read served. Additive and
serde(default, skip): old wire frames keep parsing, ordinary non-tree
traffic never carries it, and records persist as JSON payload so there is
NO schema migration. EntryTouchFold unions the touched ordinals per token;
union is commutative + idempotent, the same algebra that gives
RegionTouchFold its replay tolerance, so R-1/R-3/C-8 come for free.
mcp read --file stamps the file's ordinal; search stamps each confirmed
match's ordinal (carried on waggle_tree::Hit.entry, a stable position in
DirIndex.files()). Tree coverage replays each file-bearing node (the same
scan_token + replay path contract_coverage uses), counts
|touched ∩ [0, local_files)|, and sums across the lineage → files read /
total at true per-file granularity, with per-node unread counts and the
first missing file NAMED.
Deliberately out of scope: soa.rs (the 7-column fixed-width funnel/region
accelerator) gains no entry column — per-file coverage rides the replay path, not
the columnar one, and a u32 ordinal would muddy that structure's narrow-width
invariant for no gain.
Test rewritten: a node with two files, one read, reports "1/3" and names the file
nobody opened; a search that serves every file closes it to "3/3". Fold unit test
mirrors the region test (union commutative + duplicate-immune).
No token-per-file, so the 200-file cap stays lifted. All workspace tests pass;
clippy clean; every file <= 750 lines. Design log: waggle-tree-scale/DECISIONS.md.
…nto the CLI
The Tier-3 conformance suite (zero-LLM scripted-consumer detector) encoded the
OLD flat-tree contract — per-file child tokens, --section fan-out with a --from
cursor, a files:all gate with a met verdict. On the indexed tree it crashed
(read's `files` is now a count, not a list). Rewritten to the indexed contract it
actually ships:
projection read <root> is a table of contents — total_files (true tree size),
local files, and subdirs each with a token to descend. Asserted on
reasoning (11), folder (12), bigtree (180, nested).
per-file read <root> --file <name> serves one file and stamps exactly it:
opening 1 of 11 reports files "1/11", names the other 10 unread,
and clears its own name. This is the headline the indexing work
stands on — a receipt of WHICH file, not "the folder was touched".
search one call spans the lineage, every match names its file (path +
token); an absent literal visits ZERO nodes (Bloom prune bites);
a zero-match search moves coverage not at all.
the close a perfect consumer closes per-file coverage cheaply: one search
serves the bulk, coverage names the stragglers, read --file opens
each — reasoning closes in 2 calls.
budget the bounded responses (TOC, search) fit max-bytes.
53 checks, all green on the real corpora against the indexed binary.
Wiring gap this surfaced: read --file existed in the MCP handler but was never
exposed on the CLI — there was no command-line way to read one file of a tree, so
the per-file close path was unreachable. Added `read --file <name>` to the CLI and
the matching ops-catalog ArgSpec (parity holds).
Known follow-up (flagged, not silently changed): the READ/coverage catalog
DESCRIPTIONS still narrate the old fan-out / files:all-gate model, which the
indexed tree no longer implements — that is model-facing prose and a wording call.
Per-file coverage remains the shipped receipt; the paper's flat-tree gate/fan-out
numbers stay pinned to the pre-index backend (decision 2). All workspace tests
pass; clippy clean; every file <= 750 lines.
…sured backend Limitations gains a scope note on scale (decision: option 2). The evaluated system mints one token per file — per-file coverage for free, but bounded past a few hundred files. Subsequent work replaces the per-file forest with an indexed directory tree (content-addressed nodes, per-node trigram + Bloom): thousands of files in one mint, one search pruned and ranked across the lineage, coverage still per file but now WITHOUT a token per file — each read records the file's position in its node's signed index, a position not a payload, admissible under the same payload-free-log invariant (sec:receipts) as the region-touch field. Crucially honest: this is validated by the conformance suite (deterministic), NOT by re-running the answer-quality sweep, which is reported on the per-file system it was collected on. Whether indexing moves those numbers is future measurement, not a claim drawn from these. No results numbers change. Adds the Bloom (1970) citation. Compiles clean (tectonic) — no undefined refs or citations.
…oven per file
mint --tree --require files:all silently ignored the contract: you asked for a
gate and got none. Now the root node carries the files:all contract (tree_mint
parses it via the shared contract_args path; a folder has no single text, so a
section: requirement is rejected — only files:all is meaningful over a tree), and
tree coverage emits `met`/`requires` alongside the per-file `files: read/total`.
`met` flips false -> true as the last file is read. A plain --tree carries no
contract and reports `complete` as a fact, never a verdict.
This restores the gate the flat tree had — and per-file coverage makes it
STRONGER: met is proven file by file, not node by node. The paper's gate results
are reproducible on the shipping backend again.
Verified live on corpus2/reasoning_0 (11 files, files:all): met=false at 0/11,
met=true at 11/11 after one search + one read --file.
Also fixes a pre-existing test flake: two extraction tests shared the temp dir
name `waggle-extract-{pid}`, so in parallel one's cleanup wiped the other's blobs
(binary_target_...pdf_story failed intermittently). Given distinct html/pdf names,
the suite is green multi-threaded. All workspace tests pass; clippy clean.
…/tree, drop --from
The operations catalog (the source of truth for CLI help, MCP tool descriptions,
and COMMANDS.md) described the OLD flat model: mint --tree "mints every file as a
child", read "FANS OUT a lens across every file" with an examined/total_files
truncation and a --from cursor, search groups matches "each with that file's own
token". None of that is how the indexed tree works.
Corrected to the shipping model:
--tree builds an indexed directory tree — one content-addressed node per
folder (trigram + Bloom), thousands of files in one mint.
read a folder token returns its table of contents (files by name, subdirs
by token); read --file <name> serves one file.
search spans the whole tree in ONE call — pruned, narrowed, ranked; each
match names its file path + owning node token.
coverage a per-file receipt (files: read/total, complete, unread NAMED);
--require files:all adds the met verdict.
Removed the --from arg (CLI + catalog): the fan-out it paged no longer exists, so
it was a dead, misleading flag. --require's files:all clause is unchanged — it is
correct again now that the gate is wired. COMMANDS.md regenerated; parity green.
…sign logs
The catalog fix (prior commit) corrected the generated surfaces; this corrects the
hand-written ones to the indexed tree + per-file model.
Site (docs/pages/index.html): the tutorial's step 3 listed a token per file and
step 4 fanned a --section lens "across every file at once". Now step 3 shows the
table of contents (files by name) and points at `read --file`; step 4 is the
one-call tree `search`, pruned and ranked. Steps 2 and 5 (--require files:all,
coverage with met) are unchanged — correct now that the gate is wired.
Guides:
04-lifecycle-and-query read --section fan-out -> read --file + one-call search;
coverage key read -> files.
07-surgical-content per-child funnels -> per-file receipts; run level ->
per-file files: read/total + complete + met.
11-tmux-switchboard per-file child tokens -> indexed tree; resolve INDEX of
file-tokens -> read table of contents + read --file.
Design logs (historical decision records): added "superseded by waggle-tree-scale"
banners to 18 §9 and 22 §4 rather than erasing the fan-out history — the failures
they name were real; only the mechanics changed. Fixed 18 §9.1 (per-file token ->
file by name) and 20-symbol-lens's "≤ 200 files" -> indexed, thousands.
Core: TreeNode doc-comment claimed a per-file token is minted lazily — no such
minting exists; files are addressed by name. Corrected.
The paper stays historical (measured pre-index) and needs no change. No code
behavior changed here; workspace builds, fmt clean.
Indexed directory tree (cap lifted to thousands of files), per-file coverage receipts, files:all gate rewired onto trees, and the CLI read --file affordance. Breaking CLI change: the dead --from fan-out cursor is removed.
On Windows, a directory node's dir_url was built with dir.display() (backslash separators), and the parent derived each subdir's name by splitting that URL on '/', which returned the whole path instead of the leaf. The projection's subdir names were wrong, so a consumer could not find a subdirectory by name. Carry the real base name in NodeData (from entry.file_name(), correct on every platform) and drop the URL-parsing dir_name helper. Fixes the Windows-only content_access::coverage_is_per_file_over_a_tree failure; the other platforms were unaffected because '/' is their native separator.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Replaces the per-file-token folder model with an indexed directory tree, lifting the ~200-file mint cap to thousands of files in one call, and makes coverage a per-file receipt.
Backend (waggle-tree + mcp)
waggle-tree— new pure crate: fixed-size union-composable Bloom, trigram inverted index, MerkleDirIndex, ranking primitives.read --file <name>serves one file by name.files: read/total,complete, unread NAMED), via an additiveEvent.entryordinal (a position into the signed dir index — I-1-safe like the region-touch field) folded by a commutative/idempotentEntryTouchFold.files:all gate, rewired onto trees
mint --tree --require files:allnow attaches the contract to the root node and coverage emitsmet(false→true as the last file is read) — the flat tree's gate, now proven per file. Verified live on the reasoning corpus (0/11 → 11/11).Benchmark + docs
Breaking / release
--fromfan-out cursor is removed fromread.read --file <name>.0.4.0 → 0.5.0.Merging deploys the site (
pages.yml) and recompiles/publishes the paper PDF (paper.yml). Taggingv0.5.0cuts binaries + the Homebrew tap viarelease.yml.🤖 Generated with Claude Code