Feature/Fix: Integrate lineage discovery and shadow accounting

[iam-brain]

Stack dependency

This is PR 2 of 4. It follows PR #2155; review and merge that PR first.

Summary

Integrates bounded lineage discovery, token-accounting dimensions, and shadow execution with the existing scanner.

• pulls exceptional long-lived parents outside the normal scan window only when referenced by an included descendant
• indexes physical filename owners for parent discovery; retained metadata aliases cannot suppress the actual parent
• preserves UTC/local-day, model, cached-input, output, pricing, and physical-owner dimensions
• runs privacy-safe lineage comparison after refreshed legacy scans
• keeps the legacy report and cache authoritative
• contains non-cancellation shadow failures without changing production output

Validation

• focused discovery, ledger, and shadow suites, including retained-parent metadata regression coverage
• redacted frozen-history replay summarized in PR Feature/Fix: Validate and optimize lineage token accounting #2158
• make check
• full suite: 650 selections across 55 groups; no failures, retries, or timeouts
• repository review gate: no material findings after fixes

Related investigation: #2037.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 14, 2026, 8:23 PM ET / July 15, 2026, 00:23 UTC.

Summary
Adds bounded Codex ancestor discovery, multidimensional lineage accounting, and a contained non-authoritative shadow comparison pass to the existing scanner.

Reproducibility: yes. for the underlying fork-lineage overcount: the related investigation and frozen-corpus comparisons establish a high-confidence accounting failure. The runtime behavior and cost of this exact integration head have not been directly demonstrated.

Review metrics: 3 noteworthy metrics.

• Patch size: 8 files; 1,347 added, 12 removed. The PR introduces a substantial accounting subsystem despite leaving production totals unchanged.
• New focused suites: 3 test suites added. Discovery, ledger, and shadow behavior receive dedicated regression coverage.
• Stack position: PR 2 of 4. The branch cannot be evaluated or landed independently of the unmerged foundation and later safety and validation layers.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Provide redacted exact-head runtime output for cold, changed-input, repeated-warm, cancellation, and contained-failure scenarios.
• Rebase after Feature/Fix: Add a shadow lineage ledger for Codex token accounting #2155 and document how the resulting scanner reconciles with Fix Codex fork parent cache invalidation #2118.

Proof guidance:

• [P1] Needs stronger real behavior proof before merge: The linked frozen-history replay is useful but exercises the later combined stack rather than this exact head; add redacted terminal output or logs for cold, changed-input, repeated-warm, cancellation, and contained-failure runs, then update the PR body to trigger review or ask a maintainer for @clawsweeper re-review.

Risk before merge

• [P1] Changed Codex corpora trigger an additional lineage discovery and parsing pass; large retained histories could noticeably extend refreshes or consume resources even though redundant unchanged scans are now gated.
• [P1] The branch depends on an unmerged foundation and touches identity/parsing behavior also changed by Fix Codex fork parent cache invalidation #2118, so landing either path without reconciliation could create conflicting assumptions or duplicated maintenance.
• [P1] The available replay validates a later combined stack rather than this exact head, leaving cancellation, contained-failure, changed-input, and repeated-warm runtime behavior unproven here.

Maintainer options:

1. Prove bounded refresh behavior first (recommended)
   Rebase after the foundation, reconcile the competing cache fix, and provide exact-head cold, changed-input, repeated-warm, cancellation, and contained-failure timings before merge.
2. Accept experimental scan overhead
   Land the shadow pass while keeping legacy authoritative and explicitly accept that changed corpora may incur a second full parsing path until later optimization lands.
3. Pause the lineage stack
   Close or defer this stack if the narrower proven fix resolves the user-visible overcount without requiring a parallel accounting architecture.

Next step before merge

• [P1] A maintainer must choose the accounting architecture and stack order, then require exact-head runtime proof; there is no narrow automated repair to apply.

Maintainer decision needed

• Question: Should CodexBar adopt this four-PR lineage-ledger architecture, or prefer the narrower identity and cache-invalidation approach in Fix Codex fork parent cache invalidation #2118?
• Rationale: Both approaches address the same active overcount investigation but establish different long-term accounting boundaries; code review cannot safely choose the permanent scanner architecture or stack order without maintainer intent.
• Likely owner: steipete — This choice sets the long-term accounting architecture and integration order for the repository's central Codex scanner.
• Options:
  • Adopt the lineage stack (recommended): Merge the foundation first, reconcile the overlap with the narrower fix, and require exact-head runtime and refresh-cost proof before landing this integration layer.
  • Prefer the narrower fix: Prioritize the proven identity and cache-invalidation repair and pause this stack until remaining overcount evidence demonstrates that a lineage ledger is still necessary.
  • Request a consolidated proposal: Have the overlapping contributors produce one smaller architecture that preserves the proven cache fix and introduces shadow accounting behind a bounded integration seam.

Security
Cleared: The diff adds no dependency, workflow, secret, permission, download, or publishing changes, and the shadow diagnostics are designed to avoid emitting private rollout contents.

Review details

Best possible solution:

Keep legacy accounting authoritative, select one canonical identity/cache foundation, then land the lineage shadow path only after the stack is rebased and reconciled with competing work and this exact head demonstrates bounded cold, changed-input, warm, cancellation, and failure behavior.

Do we have a high-confidence way to reproduce the issue?

Yes for the underlying fork-lineage overcount: the related investigation and frozen-corpus comparisons establish a high-confidence accounting failure. The runtime behavior and cost of this exact integration head have not been directly demonstrated.

Is this the best way to solve the issue?

Unclear. The shadow-first design is conservative and well tested, but the four-layer architecture overlaps a narrower proof-positive repair and requires a maintainer decision about the permanent accounting boundary.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against c61e01e774c4.

Label changes

Label justifications:

• P2: This is a normal-priority accounting improvement for a real overcount investigation, but it remains shadow-only and does not yet alter production totals.
• merge-risk: 🚨 availability: Merging adds lineage discovery and reparsing work to changed Codex refreshes, with large-history runtime cost not yet proven on this exact head.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The linked frozen-history replay is useful but exercises the later combined stack rather than this exact head; add redacted terminal output or logs for cold, changed-input, repeated-warm, cancellation, and contained-failure runs, then update the PR body to trigger review or ask a maintainer for @clawsweeper re-review.

Evidence reviewed

What I checked:

• Legacy behavior remains authoritative: The patch runs lineage accounting as a shadow comparison, contains non-cancellation failures, and does not replace the legacy report or cache. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:80, eb019ba4db3a)
• Focused regression coverage: The branch adds separate discovery, ledger, and shadow suites covering transitive archived parents, physical filename ownership, accounting dimensions, duplicate suppression, malformed timestamps, and unchanged legacy totals. (Tests/CodexBarTests/CodexLineageShadowTests.swift:1, eb019ba4db3a)
• Prior performance finding addressed: The last commit gates redundant shadow scans after review identified unconditional warm-cache reparsing; the previously reviewed head and current head are both eb019ba, with no outstanding prior code finding. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:80, eb019ba4db3a)
• Exact-head proof remains insufficient: The PR links a frozen-history replay from the later full-stack PR, but does not show this exact head running cold, changed-input, repeated warm, cancellation, and contained-failure scenarios in a real setup. (eb019ba4db3a)
• Unmerged stack dependency: The PR explicitly depends on the still-open foundation at Feature/Fix: Add a shadow lineage ledger for Codex token accounting #2155 and is itself required by two later open stack layers. (eb019ba4db3a)
• Competing accounting fix: Fix Codex fork parent cache invalidation #2118 addresses canonical child identity and parent-dependent cache invalidation for the same overcount investigation with positive cold-and-warm corpus proof, so integration order and overlap need an explicit choice. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, 8152bcae8f6b)

Likely related people:

• steipete: The change introduces a substantial new accounting architecture in the central Codex scanner, making the repository owner the strongest available decision-routing candidate. (role: repository owner and likely architecture decision owner; confidence: high; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift)
• iam-brain: Beyond authoring this stack, this contributor has prior merged work in Codex cost history and model-level token accounting, including repository history associated with pull requests 546, 561, and 1070. (role: lineage accounting feature owner and prior cost-usage contributor; confidence: high; commits: 9245047c8539, 9280c367731e, eb019ba4db3a; files: Sources/CodexBarCore/Providers/Codex/CodexLineageDiscovery.swift, Sources/CodexBarCore/Providers/Codex/CodexLineageLedger.swift, Sources/CodexBarCore/Providers/Codex/CodexLineageShadow.swift)
• asonawalla: Prior merged work reduced redundant JSONL scanner passes, which is directly relevant to the extra discovery and parsing work introduced by shadow accounting. (role: recent scanner performance contributor; confidence: medium; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (20 earlier review cycles; latest 8 shown)

• reviewed 2026-07-14T22:05:24.405Z sha eb019ba :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T22:22:29.552Z sha eb019ba :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T22:38:58.353Z sha eb019ba :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T22:56:16.498Z sha eb019ba :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T23:13:41.321Z sha eb019ba :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T23:30:22.049Z sha eb019ba :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T23:48:15.806Z sha eb019ba :: needs real behavior proof before merge. :: none
• reviewed 2026-07-15T00:06:48.886Z sha eb019ba :: needs real behavior proof before merge. :: none

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d364597b2c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Avoid re-parsing cached Codex files in the shadow pass

When a Codex refresh runs after the min interval with an unchanged corpus, scanCodexFile can reuse cached per-file data, but this unconditional shadow call walks files and CodexLineageShadow.run parses every included JSONL again via CodexLineageDiscovery.discover/parseCodexLineageDocument. That reintroduces the scan-storm class covered by the performance gates for warm refreshes, and the cost scales with the entire in-window/cached corpus even though the shadow result is only logged; gate/sample it or derive it from cached rows so warm refreshes do not re-read all session files.

Useful? React with 👍 / 👎.