fix: retain incomplete JSONL tails

[ShiroKSH]

Summary

• retain incomplete JSONL tails, including oversized records that are still being written
• rebuild cost-usage cache artifacts on upgrade so old offsets are not reused
• preserve terminal records clipped by the configured size limit
• cover completed appends, valid final lines without a newline, complete and incomplete truncated EOF records, and cache upgrades

Checks

• git diff --check

[devYRPauli]

LGTM. I checked this out and verified both the bug and the fix locally.

Running the PR's new test against the pre-fix source (CostUsageJsonl.swift on main, PR test file overlaid) reproduces a real data-corruption bug: jsonl scanner retries an incomplete final record after append fails with the still-forming final record partially consumed on the first pass (firstPass = ["{\"type\":\"message\",\"id\":\"partial"]), resumeOffset advancing to 31 instead of 0, and the record's later completion mis-parsed as a bogus fragment "\"}" on the second pass. On this branch all four tests pass.

So the real-world effect being fixed is: an in-progress final JSONL line (a session record still being written) could be partially read as if complete, its committed offset moved past it, and its eventual completion then decoded as a spurious separate line - silent cost/usage miscount on incremental tailing. The committedOffset (only advanced on a newline-terminated line) plus hasCompleteJSONTail() (only flush a no-newline tail if it already parses as valid, non-truncated JSON) is the right, narrow fix.

Edge cases I looked at and am satisfied by:

• Empty file / file ending exactly on \n: hasCompleteJSONTail() no-ops, so no behavior change.
• Truncated tail: the !truncated && lineBytes == current.count guard refuses to validate a partial buffer that exceeded prefixBytes, which is the conservative choice.
• No double-emit: when the tail is committed early and the lone trailing \n arrives on a later scan, appendSegment/flushLine no-op on the zero-length segment, so the record isn't emitted twice.

Two minor, non-blocking notes:

• The second new test (accepts a complete final record without newline) passes on the pre-fix source too, so it reads as a non-regression/documentation test rather than proof of the fix - the retries... test is the one carrying the regression coverage.
• hasCompleteJSONTail() parses the tail once to validate, and the consumer parses it again when the line is finally accepted - a small redundant parse, not worth changing.

Nice catch on a subtle one.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 16, 2026, 3:29 PM ET / 19:29 UTC.

Summary
The PR preserves structurally incomplete JSONL tails for subsequent appends, commits complete clipped EOF records, invalidates stale cost-scan cache artifacts, refreshes the generated parser hash, and adds extensive regression coverage.

Reproducibility: yes. at source level: the former EOF path could not reliably distinguish complete records from still-extendable JSON tails, and the focused fixtures cover the failing append and truncation cases. A real current-installation reproduction transcript has not been supplied.

Review metrics: 3 noteworthy metrics.

• Patch surface: 8 files, +817/-16. The patch is concentrated in one scanner, cache-version constants, generated metadata, and regression coverage rather than broad product code.
• Scanner coverage: 498 test lines added. The large adversarial matrix materially raises confidence in EOF, UTF-8, scalar, clipping, and append behavior.
• Upgrade invalidation: 4 cache artifact families bumped. Codex, Claude, Vertex AI, and Pi users will receive fresh cache files and a full rescan after upgrading.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🧂 unranked krab
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Post a redacted terminal transcript demonstrating partial-write completion, resumed scanning, and exactly-once accounting on a real JSONL file.
• Show a representative predecessor-cache upgrade rebuilding safely, with private paths, identifiers, endpoints, and credentials removed.
• [P1] Report focused scanner/cache tests and the repository-required make check result in the PR body.

Proof guidance:

• [P1] Needs real behavior proof before merge: The PR body contains only git diff --check; before merge, add a redacted terminal transcript showing a real partial JSONL write, completion append, resumed scan, exactly-once result, and predecessor-cache rebuild. After updating the PR body, a fresh review should trigger automatically; otherwise ask a maintainer to comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

• [P1] Existing Codex, Claude, Vertex AI, and Pi cache artifacts will be ignored after upgrade, causing a one-time full rescan whose latency and resource impact have not been demonstrated on a representative installation.
• [P1] No contributor-owned live transcript currently proves that a partially written real session record is retained, completed, counted exactly once, and persisted with a safe resume offset.

Maintainer options:

1. Prove and accept the rebuild (recommended)
   Keep the corrected scanner and all artifact bumps, but require a representative upgrade transcript showing safe offsets and acceptable one-time rescan behavior.
2. Reduce invalidation scope
   Preserve unaffected predecessor caches if maintainers determine that some provider artifacts cannot contain offsets produced by the changed scanner.
3. Defer the compatibility cost
   Pause the PR if a full multi-provider rescan is not acceptable without a selective migration path.

Next step before merge

• [P1] Wait for contributor-owned real behavior proof and an explicit owner decision accepting or narrowing the four-artifact upgrade rebuild; there is no concrete mechanical patch defect for automation to repair.

Maintainer decision needed

• Question: Should the corrected tail semantics intentionally invalidate all Codex, Claude, Vertex AI, and Pi cost-cache artifacts during upgrade?
• Rationale: The broad version bumps prevent stale offsets from skipping retained tails, but they also impose a one-time full rescan on every affected existing installation; accepting that compatibility cost requires repository-owner intent.
• Likely owner: steipete — They authored the final scanner corrections and tests and are therefore best positioned to judge whether the broad cache invalidation is an acceptable upgrade tradeoff.
• Options:
  • Accept the full rebuild (recommended): Keep all four artifact bumps and merge after live proof demonstrates correct recovery and acceptable rebuild behavior.
  • Narrow the invalidation: Limit version bumps to artifact families proven to persist offsets affected by this scanner, with focused compatibility tests for unchanged providers.
  • Pause for migration design: Hold the PR until a more selective offset migration or repair strategy avoids discarding complete predecessor caches.

Security
Cleared: The diff adds local parsing logic, cache schema bumps, generated hash output, and tests without introducing dependencies, downloaded execution, secret access, permission changes, or release automation.

Review details

Best possible solution:

Retain the structurally aware EOF scanner and cache-version migration, then merge after a representative redacted live run proves partial-write recovery, exactly-once accounting, and acceptable one-time upgrade rebuild behavior.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: the former EOF path could not reliably distinguish complete records from still-extendable JSON tails, and the focused fixtures cover the failing append and truncation cases. A real current-installation reproduction transcript has not been supplied.

Is this the best way to solve the issue?

Yes, conditionally: structural EOF classification at the shared scanner boundary is narrower and safer than provider-specific workarounds, while cache invalidation is appropriate only if maintainers accept and prove its upgrade cost.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 1b921ecf3930.

Label changes

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P2: This is a bounded cost-accounting correctness fix with meaningful but limited user impact and no evidence of a core-runtime outage.
• merge-risk: 🚨 compatibility: Merging intentionally stops reusing four predecessor cache formats and may impose a noticeable one-time rescan on existing installations.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body contains only git diff --check; before merge, add a redacted terminal transcript showing a real partial JSONL write, completion append, resumed scan, exactly-once result, and predecessor-cache rebuild. After updating the PR body, a fresh review should trigger automatically; otherwise ask a maintainer to comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

What I checked:

• Tail-preservation implementation: The head introduces a structural JSON tail state machine so extendable EOF records can be retained without repeatedly rescanning already-classified bytes. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageJsonl.swift:9, 838e64ac66cb)
• Adversarial regression coverage: The scanner test suite adds 498 lines covering incomplete appends, newline-free final records, clipped complete and incomplete records, scalars, UTF-8 boundaries, oversized tails, and short prefixes. (Tests/CodexBarTests/CostUsageJsonlScannerTests.swift:61, 838e64ac66cb)
• Cost-cache upgrade behavior: The patch increments the Codex cache from v9 to v10 and the Claude and Vertex AI caches from v4 to v5, preventing reuse of persisted offsets produced under the old tail semantics. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageCache.swift:14, 838e64ac66cb)
• Pi cache upgrade behavior: The Pi session cache artifact version moves from v5 to v6 so old scan offsets and derived usage artifacts are rebuilt under the corrected parser behavior. (Sources/CodexBarCore/PiSessionCostCache.swift:5, 838e64ac66cb)
• Re-review continuity: The previous completed ClawSweeper cycle found no discrete patch defect at dba4ef4; subsequent commits added adversarial tests, UTF-8 and scalar validation, numeric-tail handling, state simplification, and a fresh main merge. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageJsonl.swift:9, 838e64ac66cb)
• Real behavior proof remains absent: The PR body reports only git diff --check; the supplied discussion contains no redacted live transcript, logs, recording, or artifact demonstrating a partial real JSONL write, completion append, resumed scan, and predecessor-cache rebuild. (838e64ac66cb)

Likely related people:

• steipete: They contributed the branch's adversarial tests and successive correctness refinements for UTF-8, scalar, numeric-tail, parser-state, and generated-hash behavior, making them the clearest owner for the remaining compatibility decision. (role: reviewer and recent area contributor; confidence: high; commits: 12e1239430bd, 518dd887ee77, a1d1b31ecf2a; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageJsonl.swift, Sources/CodexBarCore/Generated/CodexParserHash.generated.swift, Tests/CodexBarTests/CostUsageJsonlScannerTests.swift)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (21 earlier review cycles; latest 8 shown)

• reviewed 2026-07-14T22:39:04.837Z sha dba4ef4 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T22:55:18.051Z sha dba4ef4 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T23:11:24.028Z sha dba4ef4 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T23:27:23.264Z sha dba4ef4 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-14T23:44:55.895Z sha dba4ef4 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-15T00:03:26.982Z sha dba4ef4 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-15T00:20:11.542Z sha dba4ef4 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-15T00:47:57.873Z sha dba4ef4 :: needs real behavior proof before merge. :: none

[steipete]

Merged as 423cf7a5c6f0709512eebc5d7f69454dc0b2d08b after verifying exact PR head 838e64ac66cb5bac953d360ec36bef2688d9108d.

Verification:

• Audited the full diff and the incremental scanner state machine. Offsets now advance only at newline boundaries or at EOF when the JSON tail is structurally complete. String escapes, nested containers, literals, numbers/exponents, CRLF, trailing whitespace, and records larger than the 256 KiB read buffer are covered. State remains O(1); an incomplete large record is re-read from disk instead of accumulated in memory.
• Merged current origin/main into the contributor branch without force-pushing. Cache artifact versions are Codex 10, Claude/Vertex AI 5, and Pi 6. Parallel PR Fix copied-prefix Codex subagent accounting #2228 remained open and did not collide with this scanner change.
• swift test --filter CostUsageJsonl: 19 tests in 3 suites passed, including adversarial truncated escape, nested string delimiter, CRLF, over-buffer, literal, number, exponent, and trailing-whitespace cases. Both scanner benchmarks passed.
• make check: passed; parser hash current (577571954a2d36bc), SwiftFormat 0/1465 changes, SwiftLint 0/1464 violations.
• make test: all 659 selections across 55 groups passed after two transient full-group retries; zero final failures or timeouts. The unrelated bounded-child proof was rerun directly with swift test --filter BoundedChildProcessProofTests: 3/3 passed.
• Final autoreview: clean; no accepted or actionable findings.
• Exact-head CI: all eight checks green, including both macOS shards, both Linux CLI builds, lint, changes, lint-build-test, and GitGuardian. GitHub reported MERGEABLE / CLEAN immediately before merge.