Feature/Fix: Stream lineage families in two passes

[iam-brain]

Stack dependency

This is slice 9 of 15. It follows Feature/Fix: Bound lineage reconciliation work; review and merge that PR first.

Summary

• add metadata-only pass-one parsing that retains no token observations
• discover scoped active/archive ancestors through lightweight signed descriptors
• load, reconcile, and release one lineage family at a time
• reuse warm family cache entries without loading their documents
• reject rollout files changed during or after discovery before publishing a candidate

This completes the end-to-end memory-bound seam identified while implementing #17.

Validation

• swift test --filter 'CodexLineage(Engine|TwoPassDiscovery)Tests' (13 passed)
• make check
• env SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX26.5.sdk make test (643 selections, 54 groups, no failures or retries)

Related investigation: steipete/CodexBar#2037.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 13, 2026, 9:19 PM ET / July 14, 2026, 01:19 UTC.

Summary
The PR adds metadata-only lineage discovery and family-at-a-time reconciliation with cache reuse, file-mutation detection, parser support, shadow accounting, and focused tests.

Reproducibility: yes. at the source and fixture level: the canonical investigation and tests establish forked rollout overlap, missing cross-file provenance, and the bounded-memory requirement. The PR does not provide a real-corpus reproduction of its peak-memory claim.

Review metrics: 3 noteworthy metrics.

• Patch surface: 14 files; 3,356 added, 17 removed. The cumulative diff is large for an internal accounting seam and warrants review after its predecessor is isolated.
• Stack position: Slice 9 of 15; 9 commits. The open predecessor and six downstream slices make merge order and regression attribution important.
• Test additions: 6 new test files. The branch has substantial automated coverage despite lacking real-corpus runtime proof.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #2037
Summary: The canonical investigation tracks fork-related Codex usage inaccuracies; this PR is one internal implementation slice in the proposed fix stack rather than a standalone duplicate.

Members:

• canonical: Ultra-mode Terra and Sol sessions can overcount forked context #2037 - The investigation documents the fork-heavy usage-accounting discrepancy and the remaining cross-file provenance work.
• partial_overlap: Contain Codex interleaved overcount and add fork provenance fixtures (#2037) #2082 - This merged PR implemented the earlier bounded per-file scanner and cache foundation but intentionally left cross-file lineage provenance unresolved.
• fixed_by_candidate: Feature/Fix: Bound lineage reconciliation work #2147 - This open predecessor bounds reconciliation work and must land before the current family-streaming slice.
• adjacent_distinct: Feature/Fix: Define lineage promotion gates #2149 - This downstream slice defines promotion gates after the current two-pass implementation rather than replacing it.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• After the predecessor lands, rebase the branch so reviewers can isolate this slice's unique changes.
• [P1] Add redacted cold- and warm-scan evidence showing memory bounds, totals, cache behavior, cancellation, and mutation rejection.

Proof guidance:

• [P1] Needs real behavior proof before merge: The PR provides tests and CI-style commands only; before merge, add redacted terminal output, logs, or a recording from cold and warm scans of a real affected corpus, then update the PR body to trigger review or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

• [P1] The generated parser-hash change forces existing installations to rebuild their Codex usage cache; without a real upgrade run, peak memory, scan latency, cancellation behavior, and resulting totals on a large fork-heavy corpus remain unproven.
• [P1] The PR is based directly on main while containing nine cumulative commits and 3,356 additions from a larger stack; reviewing or merging before Feature/Fix: Bound lineage reconciliation work #2147 makes it difficult to isolate regressions to this slice.
• [P1] The new lineage path is described as shadow accounting and does not yet replace production totals, so passing unit tests alone does not demonstrate that family streaming actually bounds memory under realistic corpus size and filesystem mutation.

Maintainer options:

1. Rebase and prove the upgrade path (recommended)
   Land the predecessor, rebase this slice, and add redacted live evidence for cold cache rebuild, warm reuse, bounded memory, cancellation, and unchanged production totals before merge.
2. Accept the unproven cache rebuild
   Merge after the predecessor based on automated coverage while explicitly accepting that large-corpus upgrade performance and accounting output have not been observed in a real setup.
3. Pause the cumulative stack
   Keep the draft paused until the lineage work is reorganized into independently reviewable slices with proof attached to each runtime-affecting step.

Next step before merge

• [P1] A maintainer should enforce the stack order and review the rebased architecture after the contributor supplies real-corpus proof; there is no narrow mechanical defect for an automated repair worker.

Security
Cleared: The diff adds no dependencies, workflow permissions, secret handling, downloads, or executable supply-chain sources; local rollout contents are parsed and hashed without evidence of new disclosure.

Review details

Best possible solution:

Merge the predecessor first, rebase this slice so its unique delta is reviewable, and demonstrate initial and warm scans on a redacted real fork-heavy corpus with peak-memory diagnostics, cache-rebuild behavior, cancellation, mutation rejection, and confirmation that legacy production totals remain unchanged.

Do we have a high-confidence way to reproduce the issue?

Yes at the source and fixture level: the canonical investigation and tests establish forked rollout overlap, missing cross-file provenance, and the bounded-memory requirement. The PR does not provide a real-corpus reproduction of its peak-memory claim.

Is this the best way to solve the issue?

Unclear pending stack isolation and live proof; two-pass descriptor discovery plus one-family-at-a-time loading is a coherent narrow architecture, but the cumulative branch and unobserved upgrade behavior prevent confirming it as the best production-safe implementation.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 0437067deaa8.

Label changes

Label changes:

• add P2: This is a substantial but still shadow-only accounting improvement with limited immediate user blast radius and no demonstrated current production regression.
• add merge-risk: 🚨 compatibility: Changing the generated parser hash rebuilds existing usage caches, and the PR does not yet prove cold-upgrade behavior on a real large corpus.
• add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
• add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR provides tests and CI-style commands only; before merge, add redacted terminal output, logs, or a recording from cold and warm scans of a real affected corpus, then update the PR body to trigger review or ask a maintainer to comment @clawsweeper re-review.

Label justifications:

• P2: This is a substantial but still shadow-only accounting improvement with limited immediate user blast radius and no demonstrated current production regression.
• merge-risk: 🚨 compatibility: Changing the generated parser hash rebuilds existing usage caches, and the PR does not yet prove cold-upgrade behavior on a real large corpus.
• rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR provides tests and CI-style commands only; before merge, add redacted terminal output, logs, or a recording from cold and warm scans of a real affected corpus, then update the PR body to trigger review or ask a maintainer to comment @clawsweeper re-review.

Evidence reviewed

What I checked:

• Stack dependency: The PR is draft slice 9 of 15 and explicitly requires the still-open Feature/Fix: Bound lineage reconciliation work #2147 to be reviewed and merged first. (5c82112d95af)
• Two-pass implementation: The branch adds descriptor-only first-pass discovery, including signatures and an explicit fileChangedDuringScan failure, before loading lineage families. (Sources/CodexBarCore/Providers/Codex/CodexLineageTwoPassDiscovery.swift:1, 5c82112d95af)
• Family streaming and reuse: The new engine introduces descriptor families, input fingerprints, reusable cached family results, and diagnostics for peak family observations and cache reuse. (Sources/CodexBarCore/Providers/Codex/CodexLineageEngine.swift:1, 5c82112d95af)
• Upgrade-visible cache invalidation: The generated parser hash changes from cdef6eb9658a43e2 to 84c52421265e9fc8, causing existing Codex usage caches to be rebuilt after upgrade. (Sources/CodexBarCore/Generated/CodexParserHash.generated.swift:4, 5c82112d95af)
• Focused automated coverage: Six new test files cover lineage discovery, ledger behavior, reconciliation, residual classification, shadow comparison, bounded observation retention, warm reuse, and file changes. (Tests/CodexBarTests/CodexLineageTwoPassDiscoveryTests.swift:1, 5c82112d95af)
• Real behavior proof gap: The PR body reports focused tests, make check, and the full suite, but contains no redacted terminal output, logs, recording, or linked artifact from an after-fix scan of a real fork-heavy corpus. (5c82112d95af)

Likely related people:

• steipete: Reviewed and merged the preceding bounded Codex usage-scanner work and authored follow-up cancellation, parser-hash, cache, and scan refinements in the same path. citeturn985633view0. (role: recent area contributor and merger; confidence: high; commits: c852c1314517, cb6538b, ab65dd3; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, Sources/CodexBarCore/Generated/CodexParserHash.generated.swift)
• Zihao-Qi: Authored the merged bounded-scanning implementation that established the current accounting and parser-cache foundation this stack extends. citeturn985633view0. (role: feature introducer; confidence: high; commits: be56cc9, 783ba29, c7294c1; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, Tests/CodexBarTests/CostUsageScannerTests.swift)
• xx205: Introduced the parent-rollout baseline adjustment used by the current production accounting path, making them relevant to lineage and fork semantics. citeturn985633view0. (role: adjacent behavior contributor; confidence: medium; commits: 45b68c34ec48; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, Tests/CodexBarTests/CostUsageScannerTests.swift)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[iam-brain]

Superseded by the consolidated four-PR review stack. This slice is now included in #2157; closing this draft to keep the upstream review surface manageable.