fix(codex): one-time rebuild of fork-inflated Codex aggregates (#75)#78
Merged
Conversation
The parser fix (PR #77) only corrects go-forward parsing. Files ingested before it sit at offset = EOF and are never re-read, so their fork-inflated contribution (up to ~40x Cache Read on the local corpus) stays baked into the persisted aggregates and the dashboard. This adds the one-time rebuild that retracts that inflated history and forces a clean re-parse. migrateCodexForkOvercountRebuild (run-once via a migrations sentinel, modeled on migrateRolloutCumulativeDeltaBuckets, scoped to source=codex/every-code so no other provider is touched): - Drops the per-file cursors so the same-run parseRolloutIncremental re-parses from offset 0 with the fixed accounting (and clears the obsolete codexFork field). - Retracts the inflated hourly buckets: deletes them from the accumulator and appends zero rows to queue.jsonl. The dashboard keep-lasts per source|model|hour_start, so the re-parse's corrected row supersedes the zero, or the zero stands for a file no longer on disk. - Resets the project accumulator buckets WITHOUT writing zero rows. Files still on disk rebuild and their corrected project row wins via keep-last; a bucket whose file was deleted or whose repo moved keeps its prior queue value rather than being zeroed. Unlike the hourly path (whose orphans were already retracted by the earlier delta migration), project aggregates were never retracted before, so a zero here would be a brand-new, unrecoverable loss. Local-only: no network. The acceptance test runs cmdSync with a rejecting fetch stub and asserts zero network calls, that the seeded 8.2M-token inflated codex bucket becomes child-only (cached 80 / total 130) on the queue.jsonl read path, and that a non-codex row is byte-identical. Additional tests cover the direct migration (codex reset, other providers untouched), orphaned project buckets (queue row preserved, no loss), and sentinel idempotency. Refs #75 Co-authored-by: itarun.p <itarun.p@somapait.com>
This was referenced Jul 22, 2026
pitimon
added a commit
that referenced
this pull request
Jul 22, 2026
Ships the issue #75 Codex subagent fork-history fix that is already on main: - #77 parser fix (exclude replayed fork parent history from token accounting) - #78 one-time rebuild migration for already-inflated Codex aggregates The version bump is what lets npm-publish.yml actually publish main (the job skips while the package.json version already exists on npm). Required so the dashboard LaunchAgent and `npx @ipv9/tokentracker-cli sync` pick up the fixed parser + migration instead of the still-inflated 0.39.38. Co-authored-by: itarun.p <itarun.p@somapait.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The #75 parser fix (PR #77, merged) corrects only go-forward parsing. Files ingested before it sit at
offset = EOFand are never re-read, so their fork-inflated contribution (up to ~40x Cache Read across the local corpus) stays baked into the persisted aggregates — the dashboard keeps showing the old inflated numbers until this migration runs. This is the second half of #75.The migration
migrateCodexForkOvercountRebuild— run-once (via acursors.migrationssentinel, keycodexForkOvercountRebuild_2026_07), modeled on the adjacentmigrateRolloutCumulativeDeltaBuckets, scoped tosource=codex/every-codeso no other provider is touched. Invoked incmdSyncjust beforeparseRolloutIncremental, so the same run rebuilds corrected numbers.codexForkcursor field).queue.jsonl. The dashboard keep-lasts persource|model|hour_start(readQueueData), so the re-parse's corrected row supersedes the zero — or the zero stands for a file no longer on disk.Local-only (no network)
The acceptance test runs
cmdSyncwith a rejectingfetchstub and asserts zero network calls, satisfying #75's local-only requirement.Review
Two independent reviews (code-reviewer + silent-failure-hunter). code-reviewer: no CRITICAL/HIGH — retraction identity, schema, scoping, and read-path test all verified against primary sources. silent-failure-hunter raised one HIGH (the project zero-retraction was a new data-loss surface for orphaned files); fixed by the conservative accumulator-only reset above, with an orphan-preservation test.
Test plan
test/sync-codex-fork-migration.test.js— 4 tests:project.queue.jsonlrow is byte-identical (no loss)cmdSync+ rejecting fetch stub → 0 network calls; seeded 8.2M-token inflated codex bucket becomes child-only (cached 80/total 130) on thequeue.jsonlread path; a non-codex row survives byte-identicalnpm run ci:localgreen (root suite 765/765)With this merged, #75 is fully resolved (parser fix + historical rebuild) and can be closed.
Refs #75