[v0.17 CAP-1] Oversized-source exclusions never reach the evidence surface

[TheGreenCedar]

Context

DEFAULT_SOURCE_FILE_BYTE_CAP is 1,000,000 bytes (crates/codestory-contracts/src/workspace.rs:7). Any source above it is excluded from indexing as an OversizedSourceExclusionCandidate and published as a policy-exclusion generation.

That exclusion never reaches the evidence surface. FileCoverageReason::Oversized exists and has a message ("source exceeds the configured indexing size limit", index_coverage.rs:90) and a code (source_oversized, :368) — but no production path constructs one. The only construction in the tree is a test fixture (services.rs:4688). Grepping crates/codestory-runtime/src/agent/ for exclusion returns nothing: the packet layer has no knowledge of excluded files at all.

So on a repository containing a 1.2 MB source file, that file is never read, and a packet answering a question about it can still report sufficient with no gap, no typed cause, and nothing in the CLI or MCP surface telling the user a file was skipped. The exclusion is durably recorded in the store and invisible to the answer.

This is the release's own thesis inverted. Milestone 14 states: "CodeStory stops overstating its evidence: a sufficient packet now means every claim is proven." An unread file that no verdict mentions is an unproven claim by construction.

Why the cap value is the smaller half of this

1 MB is low for real codebases — generated clients, lockfile-adjacent sources, vendored single-file libraries and large hand-written modules routinely exceed it. (This very repository hit it: codestory-indexer/src/lib.rs reached 987 KB, see #1801.) The cap is env-configurable via SourceIndexPolicy, so a user can raise it — but only if they know a file was skipped, which today they do not.

Raising the number narrows the window. Surfacing the exclusion closes the honesty gap at any cap.

Contract

• An oversized exclusion produces a typed FileCoverageReason::Oversized gap on the packet coverage surface, and caps sufficiency the way other unproven-coverage causes do.
• Status/doctor name skipped files and the cap that skipped them, so raising the cap is an informed choice rather than a guess.
• The default cap is re-derived from evidence about real repository file-size distributions rather than left at a round number, and the rationale is recorded.

Done when

• A repository with a file above the cap cannot produce a sufficient packet that depends on that file without a typed oversized gap, proven by a test that fails without the change.

Refs #1801. Refs #1179.

[TheGreenCedar]

Correction to the original report. My first survey grepped codestory-runtime, codestory-workspace and codestory-store but omitted codestory-indexer, and that omission made the claim "no production path constructs one" wrong. The indexer does construct it, in three places — crates/codestory-indexer/src/lib.rs:2946, :3275, :3756 — on the fallback-guard error path, with the message "Skipped oversized source file … N bytes exceeds M byte cap".

The narrower claim survives and is the one that matters: the coverage reason never reaches the packet surface. Grepping crates/codestory-runtime/src/agent/ for coverage_reason returns nothing, and for exclusion returns nothing. The reason is produced, recorded in ErrorInfo, and consumed by index_coverage.rs for status/refresh messaging — but the packet layer, where sufficiency is decided, has no view of it.

There are also two distinct mechanisms, worth separating in whatever fixes this:

1. Planning-time exclusion — OversizedSourceExclusionCandidate, published as a policy-exclusion generation. The file never reaches the indexer.
2. Indexer fallback guard — FileCoverageReason::Oversized on ErrorInfo when a file does reach the indexer over the cap.

Both end with an unread file; neither ends with a packet that knows.

On the cap's purpose: the constant documents itself as a parser input bound ("Parser input bound shared by workspace planning and the indexer fallback guard"), with a companion DEFAULT_STRUCTURAL_UNIT_CAP = 2_048 for structural-unit explosion. That is a real resource constraint on tree-sitter parsing and structural collection, not an embedding-dimension one — embedding dimension is fixed by the model; what scales with file size is unit and chunk count. So raising the byte cap is not free, and the honest fix is likely to surface the exclusion and consider chunking or partial projection for large files rather than simply parsing them whole.

[TheGreenCedar]

Measurement results: the byte cap does not bound cost, and raising it alone is a regression

Measured on an M5 (10 cores, 24 GB), release build, via a harness calling the real index_file. Two findings change what this issue should ask for.

1. Oversized aborts the whole refresh — so a one-constant cap raise breaks indexing

index_full.rs:174 (and index_incremental.rs:333,:619) filter out only ParserPartial. Every other coverage reason becomes a blocking gap and raises source_coverage_failure / source_oversized. The indexer guard is not a quiet second skip path.

That matters because MAX_STRUCTURAL_SOURCE_BYTES = 1024 * 1024 (indexer/src/structural/mod.rs:38) is a hard literal, not policy-derived, checked at lib.rs:3283. Today the 1,000,000 policy cap fires first, so it is unreachable. Set the policy to 2,000,000 and the window 1,048,576 < size <= 2,000,000 opens: a 1.5 MB JSON/YAML/Markdown clears planning, hits the literal, and fails the entire index. Any cap change must derive that constant from SourceIndexPolicy in the same commit, and bump REPO_TEXT_MAX_FILE_BYTES (runtime/src/repo_text.rs:16) too, or 1–2 MB files become indexed-but-unsearchable.

2. Cost is a function of shape, not size — by 110x

At a fixed ~500 KB, varying only statements-per-function, index_file ranged 1,219 ms to 134,467 ms. A 500 KB single-function Rust file costs 125.6 s; the real 9.4 MB sqlite3.c parses in 401 ms and indexes in 14.5 s. The most expensive file measured anywhere is half the current cap and is admitted today. The byte cap has never bounded cost.

Parsing is not the problem: tree-sitter runs at ~19 MB/s and is exactly linear (2.03x per doubling). At 10 MB Rust, parse + TSG is 3.0 s of a 362 s total. 99.2% of the time is four quadratic loops, all fixable:

	Site	Cost	Fix
a	source_line — source.lines().nth() per node (lib.rs:4188, called from infer_access_from_source)	66% of Rust, 36% of TS	build the line-start index once per file
b	apply_rust_receiver_call_hints — literal nested loop (lib.rs:5238)	20% of Rust	HashMap on (name, line, col); ~5 lines
c	rust_enclosing_value_types re-walks the enclosing fn per call site (lib.rs:5409)	the entire 110x shape blowup	one descending pass
d	callable_occurrence_projection_parts rescans all occurrences per callable (lib.rs:21140)	5.5% at 2 MB; 12.4B comparisons at 10 MB	sort once, binary-search the line window

Memory is linear but heavy: ~460 bytes of peak RSS per source byte (2 MB -> 753 MB, 10 MB -> 4.6 GB), and files index under rayon par_iter with an 8 MiB chunk budget, so several can be resident at once.

Recommendation

1. Raise the cap to 2,000,000 as a headroom number, not a cost bound — and say so. Justification is admission, not time: the largest hand-written file found across six checkouts is this repo's own codestory-indexer/src/lib.rs at 95–99% of the cap (see [v0.17 IDX-SPLIT] Split the indexer god file before it crosses its own oversized-source cap #1801). Across 204 local caches, 53 of 54 single-exclusion caches excluded codestory-runtime/src/lib.rs at 1.03–1.31 MB. Zero files in five real repos exceed 2 MB. Raising 1 MB -> 2 MB changes admission for exactly 7 files, all binary images.
2. Filter by type before size. A .png is not "oversized source", it is not source. Moving the language/support-mode lookup ahead of the byte-cap test removes all 7 of those exclusions.
3. Bound cost per callable body, not per file. Always parse (1.4–2.5% of cost) and always keep every declaration, so canonical ids, symbol keys and callable_signature_hash stay byte-identical to a full index; bound only body projection, and never with a cumulative per-file budget — a boundary that moves changes the callable population, classify_projection_update routes the file to FullReplace, and every bookmark in it dies. Report it as ParserPartial, which already exists and is already non-fatal.
4. This issue's own ask stands and gets sharper: runtime/src/agent/ has zero references to coverage_reason or exclusion, so an incompletely indexed file is invisible to the agent. Mirror EV-78's split exactly — exclusions that nothing in the packet touches must not cap sufficiency (that would permanently lock large repos out), but a claim resolving to an excluded path must.

One correction to my earlier framing here: DEFAULT_STRUCTURAL_UNIT_CAP = 2,048 does not apply to parser-backed source at all — it is only checked in index_structural_source_with_unit_cap, i.e. JSON/YAML/TOML/HTML/CSS/SQL/Markdown. It does, though, already discriminate data dumps from code without any extension guessing: a 139 KB JSON array produced 5,816 units and was rejected.

[TheGreenCedar]

Split the performance half of the measurement out into #1820 — the four quadratic loops are a distinct workstream from this issue's evidence-surface gap, and they are what determines whether any byte cap can be described as a cost bound (it cannot, today).

This issue keeps: the cap raise itself, deriving MAX_STRUCTURAL_SOURCE_BYTES from SourceIndexPolicy so the two limits cannot disagree, filtering by type before size, and getting exclusions onto the packet/answer evidence surface.

[TheGreenCedar]

The cap half landed on dev/codestory-next as dc063251 (PR #1821, closing #1822). DEFAULT_SOURCE_FILE_BYTE_CAP is 2,000,000; structural formats keep a separate 1 MiB bound carried on SourceIndexPolicy.

What this issue still owns: the evidence surface — crates/codestory-runtime/src/agent/ has zero references to coverage_reason or exclusions, so an incompletely indexed file remains invisible to the agent. Mirror EV-78's split: an exclusion no claim touches must not cap sufficiency (capping there would permanently lock large repositories out of packet and search), but a claim resolving to an excluded path must.

Also still owned here: filtering by type before size, so a .png stops being reported as "oversized source" — that alone removed all 7 policy exclusions across five real repositories in the measurement.

Two things learned landing the cap, both worth knowing before the rest:

1. Adversarial review found a blocking regression the change itself introduced: stage_semantic_projection_publication rebinds pinned exclusion rows to the active policy and stamped the parser headroom on every one. Harmless while all rows carried the headroom; once a structural row names its own smaller cap, the row stops qualifying and republish-projections fails permanently. Anything that rewrites an exclusion row must now use effective_byte_cap for that row's own path.

2. My earlier claim in this thread that the text lanes were fixed was overstated. They derive from the default constant, but the indexer gates on SourceIndexPolicy.byte_cap, which CODESTORY_INDEX_SOURCE_FILE_BYTE_CAP raises with no upper clamp — so the operator-override case CR-068 named is still open. Tracked separately as [v0.17 CAP-1c] Text lanes bound files by a compile-time constant, not the active source policy #1823.

[TheGreenCedar]

Correction: the hole is not where this issue says it is

I mapped the freshness precedent (EV-78) to mirror it for coverage, and the mapping refuted my own framing. There are two routes by which a packet can rest on an excluded file, and only one of them is uncapped.

Route A — an explicitly probed excluded path — already caps today. exact_path_probe_citation sets eligible_for_sufficiency: Some(false) (agent/packet_probe.rs:200), packet_missing_exact_path_claims requires citation_sufficiency_eligible (packet_sufficiency.rs:2013), and the verdict hits !input.missing_exact_path_claims.is_empty() at packet_sufficiency.rs:625. The result is already Partial. What is wrong on Route A is the reason: packet_sufficiency.rs:1358 reads as a retrieval miss, so an operator is told the file wasn't found when it was deliberately refused.

Route B is the real hole. maybe_append_required_file_scoped_source_citations mints eligible_for_sufficiency: Some(!explicit) (orchestrator.rs:3084). A file read straight off disk — behind the hardcoded metadata.len() > 1_500_000 guard at orchestrator.rs:3023, with siblings at :1326 and :2836 — that was not explicitly probed is sufficiency-eligible, can carry a proof-bearing claim, and yields Sufficient over a file the index deliberately refused. No probe status, no missing_exact_path_claims entry, nothing.

So the fix must be driven by cited paths, not probe paths. A probe-side fix — the obvious place, and where I would have started — leaves Route B untouched.

(For the record: these 1_500_000 bounds are unrelated to the cap raise in #1821. Those lanes never consult the index; they walk the filesystem and gate on metadata().len(), so a 1.6 MB file behaved identically before and after. Adversarial review confirmed that. Route B is a pre-existing evidence hole, not a regression.)

Two design points worth fixing in the plan, both from the precedent

Freshness carries two cause enums, not one. IndexFreshnessNotCheckedCauseDto says what the check did; FreshnessUnknownCauseDto says what a consumer may conclude, and it has variants the producer cannot emit (CauseUnreported, ObservationUnavailable). They are joined by one total mapping, for_observation, living in contracts so readiness and packet sufficiency cannot drift. Coverage needs the same asymmetry.

But it needs one deliberate divergence, and getting it backwards is the whole risk. Freshness's mapping takes Option<&Dto> because absence of an observation is itself unknown-and-therefore-capping. Coverage's must not: an empty observation list means "no path was checked", which must cap nothing. Lookup failure is carried per-path as a typed NotEstablished instead.

That inverts the reviewer's instinct. A reviewer pattern-matching on freshness will ask for the opt-in helper that fresh_index_observation() provides — and be wrong. It also means no existing fixture breaks, which looks like the change is inert when it isn't.

The silent no-op to guard against

Path matching. Exclusion rows store a workspace-relative normalized_path; citations carry absolute paths. A string comparison, or normalize_path_key, matches on some platforms and silently never matches on others — the plumbing would be complete, every test green, and nothing would ever cap. The comparison has to go through codestory_workspace::same_workspace_path, which packet_paths_match_exact_probe already uses (packet_sufficiency.rs:2094), and the test for it is a case-differing request against a stored exclusion.