Feature/Fix: Validate and optimize lineage token accounting

[iam-brain]

Stack dependency

This is PR 4 of 4. It follows PR #2157; review and merge that PR first.

Summary

Adds the final validation, diagnostic, and performance slice for the shadow-lineage rearchitecture.

• provides an opt-in aggregate-only replay over frozen local histories
• diagnoses reset-epoch and post-branch fingerprint theories without feeding speculative estimates into accounting
• records stage-level timing and bounded family metrics
• removes repeated copy-on-write growth and redundant timestamp parsing
• hashes JSONL during parsing to reduce mutation-check reads
• adds a bounded two-million-observation regression test
• keeps legacy removal blocked and the rollback path intact

The scanner is materially closer to OpenAI's finalized UTC totals, but it is not exact and further real-world validation remains necessary.

Final UTC comparison

Positive percentages are overcounts versus OpenAI; negative percentages are undercounts.

UTC day	Live app cache	vs OpenAI	Partial fix	vs OpenAI	Reviewed scanner	vs OpenAI	OpenAI official
2026-07-09	976,079,823	+14.47%	764,026,920	-10.40%	808,692,108	-5.16%	852,682,935
2026-07-10	7,456,347,992	+371.86%	1,438,632,374	-8.96%	1,625,777,248	+2.88%	1,580,199,588
2026-07-11	98,382,045,260	+7,708.15%	846,608,173	-32.81%	1,375,236,113	+9.15%	1,259,990,957
Aggregate	106,814,473,075	+2,792.45%	3,049,267,467	-17.43%	3,809,705,469	+3.16%	3,692,873,480

Sources and interpretation

• Live app cache: historical totals retained by the installed CodexBar cache, not a recomputed legacy scan.
• Partial fix: scanner after the interleaved-cost containment fix already landed upstream.
• Reviewed scanner: the lineage selector after addressing review feedback with copy-stable event identity and physical-parent discovery.
• OpenAI official: finalized UTC daily totals used during validation.

The aggregate is much closer than the partial fix, but day-level residuals remain significant. This does not claim exact parity or support removing the legacy fallback.

Performance

The cold retained-data replay used the same 616-document, 918,677-observation corpus for both paths.

Calculation path	Total	Change vs legacy
Legacy scanner	212.269 s	baseline
Reviewed lineage scanner	182.249 s	-30.020 s (-14.14%)

The lineage time breaks down as follows (minor rounding and timer-boundary overhead account for the remainder):

Lineage phase	Time	Share of lineage total
Bounded discovery, including referenced-parent lookup	82.903 s	45.49%
Ledger document loading and hashing	81.754 s	44.86%
Family reconciliation and deduplication	17.512 s	9.61%
Family preparation, composition, and timer overhead	0.080 s	0.04%

The ledger held at most one family at a time: 39 families total, 355,168 observations in the largest family, and 856,177 duplicate observations suppressed. This shows that discovery and repeated JSONL loading—not the lineage calculation itself—are the remaining performance targets.

Validation

• opt-in frozen UTC replay with aggregate-only output
• two-million-observation regression test
• 48 focused lineage tests after review fixes
• make check
• full suite: 650 selections across 55 groups; no failures, retries, or timeouts
• repository review gate: no material findings after fixes

Related investigation: #2037.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 14, 2026, 8:47 PM ET / July 15, 2026, 00:47 UTC.

Summary
The branch adds a reversible lineage-based Codex token-accounting pipeline with discovery, reconciliation, diagnostics, promotion gating, cache integration, frozen-history replay, performance instrumentation, and focused regression coverage while retaining legacy accounting as the default.

Reproducibility: yes. at source and real-corpus level: the linked investigation and frozen replay demonstrate forked cumulative snapshots producing extreme overcounts, but the private corpus was not independently executed in this review.

Review metrics: 3 noteworthy metrics.

• Patch surface: 24 files, +6,201/-22. This is a substantial central-accounting change despite preserving the legacy default.
• Code and test split: 13 source files, 11 test files. The branch pairs the new accounting architecture with a comparably broad focused-test surface.
• Validation corpus: 616 documents, 918,677 observations. The real frozen-history replay provides unusually strong scale and performance evidence for an external PR.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #2037
Summary: The open issue is the canonical report; this PR and the related open PRs are competing or stacked candidate fixes for its forked-context overcount.

Members:

• canonical: Ultra-mode Terra and Sol sessions can overcount forked context #2037 - It documents the observable Ultra-mode fork overcount that motivates the accounting changes.
• fixed_by_candidate: Feature/Fix: Add a shadow lineage ledger for Codex token accounting #2155 - It is the first required layer of this candidate architecture and adds the lineage-ledger foundation.
• fixed_by_candidate: Feature/Fix: Integrate lineage discovery and shadow accounting #2156 - It is the second required layer and integrates bounded discovery and shadow execution.
• fixed_by_candidate: Feature/Fix: Add conservative lineage safety and promotion controls #2157 - It is the immediate prerequisite and adds promotion, containment, and rollback controls.
• fixed_by_candidate: Fix Codex fork parent cache invalidation #2118 - It is a narrower candidate fix for fork identity and parent-cache invalidation under the same canonical investigation.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

• [P1] Merging this branch requires first landing and validating Feature/Fix: Add a shadow lineage ledger for Codex token accounting #2155, Feature/Fix: Integrate lineage discovery and shadow accounting #2156, and Feature/Fix: Add conservative lineage safety and promotion controls #2157; it is not a standalone landing candidate.
• [P1] If maintainers later authorize lineage totals, existing persisted cost history can change materially, and the available three-day corpus still shows -5.16% to +9.15% daily variance from finalized OpenAI totals.
• [P1] The repository needs one canonical direction between this broad lineage architecture and the narrower overlapping candidate at Fix Codex fork parent cache invalidation #2118 to avoid maintaining competing fork-accounting sources of truth.

Maintainer options:

1. Land as shadow-only architecture (recommended)
   Accept the added maintenance surface only after the three prerequisite PRs are reviewed, while keeping legacy totals authoritative and promotion unavailable by default.
2. Choose the narrower accounting fix
   Pause or close this stack if Fix Codex fork parent cache invalidation #2118 solves the canonical overcount without requiring a second accounting engine.
3. Require broader upgrade proof
   Before any authority promotion, add evidence covering fresh caches, upgraded caches, authorization removal, retained-window narrowing, and more finalized UTC days.

Next step before merge

• [P2] A history-connected repository owner must choose the canonical accounting architecture and stack landing order; no narrow mechanical defect remains for an automated repair worker.

Maintainer decision needed

• Question: Should CodexBar adopt the four-PR lineage stack as the canonical long-term accounting architecture, or prefer the narrower fork identity and cache-invalidation path in Fix Codex fork parent cache invalidation #2118?
• Rationale: Both paths address the same user-visible overcount through materially different maintenance boundaries, and source review cannot decide which long-term accounting model the repository wants to own.
• Likely owner: ratulsarna — Their merged work defines the current scanner's fork, cache, and refresh behavior, making them the strongest available history-based owner for the architectural choice.
• Options:
  • Adopt the shadow lineage stack (recommended): Review and land Feature/Fix: Add a shadow lineage ledger for Codex token accounting #2155 through Feature/Fix: Validate and optimize lineage token accounting #2158 in order, retain legacy authority, and defer promotion until broader parity evidence exists.
  • Prefer the narrower fix: Land or refine Fix Codex fork parent cache invalidation #2118 and pause this stack unless remaining failures prove the smaller correction insufficient.
  • Pause both paths: Keep the canonical investigation open while collecting more finalized daily totals and upgrade-cache evidence before choosing an architecture.

Security
Cleared: The patch changes local parsing and cache execution but introduces no dependency, workflow, secret, permission, publishing, or downloaded-code supply-chain concern.

Review details

Best possible solution:

Keep legacy accounting authoritative, adopt the lineage components only as a reversible shadow and diagnostic foundation if maintainers endorse the architecture, and require broader finalized-day and upgrade/cache-transition proof before enabling production authority.

Do we have a high-confidence way to reproduce the issue?

Yes at source and real-corpus level: the linked investigation and frozen replay demonstrate forked cumulative snapshots producing extreme overcounts, but the private corpus was not independently executed in this review.

Is this the best way to solve the issue?

Unclear: the implementation is careful, reversible, and well tested, but a narrower open fix targets the same root problem, so maintainers must choose the canonical maintenance boundary before merge.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against c61e01e774c4.

Label changes

Label justifications:

• P2: The PR addresses a serious but bounded local Codex cost-accounting error with a reversible, non-default architecture and no current production outage.
• merge-risk: 🚨 compatibility: Authorized lineage accounting can materially rewrite persisted cost totals and cache behavior, so upgrade and rollback compatibility require explicit maintainer ownership.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• feature: ✨ showcase: ClawSweeper spotlight: unusually compelling feature idea for maintainer attention. The idea combines materially better real-world accounting with bounded lineage reconciliation, negative-result diagnostics, explicit promotion evidence, and a preserved rollback path.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR provides after-fix redacted live replay output and performance measurements over a frozen real corpus, showing materially improved aggregate accuracy and bounded execution without exposing private paths or identities.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR provides after-fix redacted live replay output and performance measurements over a frozen real corpus, showing materially improved aggregate accuracy and bounded execution without exposing private paths or identities.

Evidence reviewed

What I checked:

• Current behavior boundary: The current default branch still documents and implements the existing local JSONL cost scanner; the lineage types are new files in this branch rather than behavior already present on main. (docs/codex.md:91, c61e01e774c4)
• Reversible authority gate: The new selector defines legacy as the default mode and only returns lineage-authoritative totals when lineage mode also carries an opaque promotion authorization. (Sources/CodexBarCore/Providers/Codex/CodexLineageAccountingSelector.swift:3, ab229362f651)
• Prior review continuity: The earlier authorization-cache and retained-window findings were followed by commits c36f341 and ab22936, and the latest completed ClawSweeper cycle reported no remaining findings. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:28, ab229362f651)
• Real-corpus validation: The PR reports the reviewed scanner at +3.16% aggregate error versus finalized OpenAI UTC totals across three days, with individual days ranging from -5.16% to +9.15%, and explicitly keeps legacy removal blocked. (Tests/CodexBarTests/CodexLineageLocalValidationTests.swift:8, ab229362f651)
• Performance and scale proof: The same 616-document, 918,677-observation frozen corpus completed about 14.14% faster than legacy, and the branch adds a bounded two-million-observation regression path plus stage and family metrics. (Tests/CodexBarTests/CodexLineageLocalValidationTests.swift:8, ab229362f651)
• Stack dependency: This PR declares itself the fourth layer and requires the still-open ledger, shadow-integration, and promotion-control PRs to land first. (ab229362f651)

Likely related people:

• ratulsarna: Merged the current scanner hardening in Harden Codex cost scanner fork and refresh handling #698 and carried the refresh, cache-reuse, and session-discovery changes that this architecture would extend. (role: recent area contributor and merger; confidence: high; commits: 7abea24, 9255100, fd19445; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, Tests/CodexBarTests/CostUsageScannerBreakdownTests.swift)
• xx205: Authored the original fork replay fix incorporated into the merged scanner hardening and now has the narrower open fork-identity and cache-invalidation candidate for the same investigation. (role: introduced current fork replay behavior; confidence: high; commits: d058431; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift, Tests/CodexBarTests/CostUsageScannerBreakdownTests.swift, Tests/CodexBarTests/CostUsageCacheTests.swift)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (25 earlier review cycles; latest 8 shown)

• reviewed 2026-07-14T22:16:54.355Z sha ab22936 :: needs maintainer review before merge. :: none
• reviewed 2026-07-14T22:32:15.978Z sha ab22936 :: needs maintainer review before merge. :: none
• reviewed 2026-07-14T22:51:11.736Z sha ab22936 :: needs maintainer review before merge. :: none
• reviewed 2026-07-14T23:07:55.939Z sha ab22936 :: needs maintainer review before merge. :: none
• reviewed 2026-07-14T23:24:33.167Z sha ab22936 :: needs maintainer review before merge. :: none
• reviewed 2026-07-14T23:40:48.480Z sha ab22936 :: needs maintainer review before merge. :: none
• reviewed 2026-07-15T00:00:22.088Z sha ab22936 :: needs maintainer review before merge. :: none
• reviewed 2026-07-15T00:18:03.245Z sha ab22936 :: needs maintainer review before merge. :: none

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b1cb89f8b1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Include authorization state in lineage cache identity

When .lineage is run with a promotion authorization, the promoted lineage totals are saved under the same producer key used for .lineage without authorization. If authorization is later removed, makeCodexRefreshPlan can decide shouldRefresh == false, so the cached lineage-authoritative days are returned without re-running select and the reversible switch does not fall back to legacy totals until a refresh or force rescan. Include the authorization state in the cache identity or force a refresh when it changes.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve retained days when publishing lineage totals

In authorized .lineage mode, selection.days is computed only from files, which is limited to the current scan window, but just above this block the cache may intentionally retain a wider retainedSinceKey/retainedUntilKey. A refresh for a narrowed date range can therefore replace the whole cache with only the narrowed window while still saving the wider scan bounds, causing later queries inside that retained window to return missing lineage history instead of triggering a rebuild. Recompute over the retained window or merge unchanged retained days when publishing lineage authority.

Useful? React with 👍 / 👎.