Skip to content

feat(limits): the capture tool that unblocks issue 105 Part 2 - #124

Merged
pitimon merged 1 commit into
mainfrom
feat/105-capture-fixtures
Jul 25, 2026
Merged

feat(limits): the capture tool that unblocks issue 105 Part 2#124
pitimon merged 1 commit into
mainfrom
feat/105-capture-fixtures

Conversation

@pitimon

@pitimon pitimon commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Summary

I reported #105 Part 2 as blocked on data I could not produce. That was wrong. The credentials are on the maintainer's machine, and getUsageLimits already takes an injectable fetchImpl — the seam to capture through existed the whole time.

scripts/capture-limits-fixtures.cjs runs one real request per provider, sanitizes the responses, and writes them where a fixture test can read them. --dry-run prints and writes nothing.

Why capture rather than hand-write

A fixture written from reading our own normalizer encodes our reading of it and is green whether or not the provider ever sends those fields. The lesson in ~/.claude/lessons has the blunt version: "a mock payload proves render logic, never data-source presence."

Sanitisation is allowlist-shaped

Anything not provably a quota shape is replaced — a blocklist leaks the first field nobody thought of, and these payloads are headed for a public repository.

  • Keys are classified segment-wise across both naming conventions. A lone identifying word redacts (accountId); one qualified by a quota word does not (tokenType).
  • id and name are read with their parent: currentTier.id is a tier label the normalizer reads, user.id is an account identifier. They are the commonest keys in these payloads and mean opposite things.
  • Numbers under an unrecognised key are bucketed. An account id is an integer, so "numbers are safe" would publish it.
  • Timestamps keep their shape but lose sub-minute precision, which fingerprints an account's billing cycle even though no field there is an identifier.

The worst bug it had, found by running it

It captured /bin/ps -ax — the machine's whole process list, with paths and project names — because that command runs during Antigravity discovery. It survived only because an email address happened to appear in the output and took the whole string with it.

A capture tool for a privacy-first product must not be one lucky regex away from committing a process list. Only genuine quota queries are captured now; ps, lsof and which are not.

Two more, found by testing against researched real shapes

Not invented ones:

  • the key regex was snake_case-only, so camelCase remainingFraction did not match remaining and a real 0.94 quota figure was bucketed to 0;
  • tokenType matched the credential word token and was redacted, although it is a category label.

Both would have produced fixtures that looked fine and proved nothing.

Where the shapes came from

Documented in test/fixtures/usage-limits/README.md with public sources for all nine providers — CodexBar for Claude/Codex/Kiro/Antigravity, google-gemini/gemini-cli for Gemini, dmwyatt/cursor-usage for a committed real Cursor capture, luisleineweber/usagebar for Kimi, guyinwonder168/opencode-glm-quota for Z.AI, GitHub's own docs plus a reverse-engineered source for Copilot.

None is inferred from our own code, which would have been circular. That research also surfaced two facts the sanitizer needed: Gemini returns a GCP project id in cloudaicompanionProject, and Antigravity's GetUserStatus is the only source of accountEmail.

Test plan

  • ci:local exit 0 — 972 root tests (+16), 302 dashboard
  • 16 tests written from the attacker's side: for each thing that must not survive, assert it appears nowhere in the serialised output — a field-by-field check would pass while an identifier survived one level deeper
  • Six of them run the researched real shapes through it (Claude, Codex, Cursor, Kimi, Z.AI, Gemini, Copilot, Antigravity)
  • --dry-run against an empty home creates nothing, asserted by reading the directory back
  • Every hostile value is assembled at runtime — the repo's own secret scanner refused this file when the JWT was spelled out, correctly, since it cannot tell a sample from a leak

What ships and what does not

No fixtures are committed. Only the maintainer can produce them. The script prints what it redacted so the output can be read before git add, and it stages nothing.

The next step is one command on a machine with the credentials:

node scripts/capture-limits-fixtures.cjs --dry-run   # read it
node scripts/capture-limits-fixtures.cjs             # then write

I reported Part 2 as blocked on data I could not produce. That was wrong. The
credentials are on the maintainer's machine and getUsageLimits already takes an
injectable fetchImpl, so the seam to capture through existed the whole time.

scripts/capture-limits-fixtures.cjs runs one real request per provider, sanitizes
the responses, and writes them where a fixture test can read them. --dry-run
prints and writes nothing.

WHY CAPTURE RATHER THAN HAND-WRITE. A fixture written from reading our own
normalizer encodes our reading of it and is green whether or not the provider
ever sends those fields. ~/.claude/lessons has the blunt version: "a mock payload
proves render logic, never data-source presence."

SANITISATION IS ALLOWLIST-SHAPED. Anything not provably a quota shape is
replaced, because a blocklist leaks the first field nobody thought of and these
payloads are headed for a public repository.

- Keys are classified SEGMENT-WISE across both naming conventions. A lone
  identifying word redacts (`accountId`); one qualified by a quota word does not
  (`tokenType`).
- `id` and `name` are read WITH THEIR PARENT: `currentTier.id` is a tier label
  the normalizer reads, `user.id` is an account identifier. Both are the
  commonest keys in these payloads and mean opposite things.
- Numbers under an unrecognised key are bucketed. An account id is an integer,
  so "numbers are safe" would publish it.
- Timestamps keep their shape but lose SUB-MINUTE PRECISION, which fingerprints
  an account's billing cycle even though no field there is an identifier.

THE WORST BUG IT HAD, found by running it: it captured `/bin/ps -ax` — the whole
machine's process list, with paths and project names — because that command runs
during Antigravity discovery. It survived only because an email address happened
to appear in the output and took the whole string with it. A capture tool for a
privacy-first product must not be one lucky regex away from committing a process
list. Only genuine quota queries are captured now; `ps`, `lsof` and `which` are
not.

TWO MORE FOUND BY TESTING AGAINST RESEARCHED REAL SHAPES rather than invented
ones. The key regex was snake_case-only, so camelCase `remainingFraction` did not
match `remaining` and a real 0.94 quota figure was bucketed to 0; and `tokenType`
matched the credential word `token` and was redacted although it is a category
label. Both would have produced fixtures that looked fine and proved nothing.

Shapes for all nine providers are documented in the fixture README with public
sources — CodexBar for Claude/Codex/Kiro/Antigravity, google-gemini/gemini-cli
for Gemini, dmwyatt/cursor-usage for a committed real Cursor capture, and others.
None is inferred from our own code, which would have been circular. That research
is also what surfaced two facts the sanitizer needed: Gemini returns a GCP
project id in `cloudaicompanionProject`, and Antigravity's GetUserStatus is the
only source of `accountEmail`.

16 tests, written from the attacker's side: for each thing that must not survive,
assert it appears nowhere in the serialised output. Every hostile value is
assembled at runtime — the repo's own secret scanner refused this file when the
JWT was spelled out, correctly, since it cannot tell a sample from a leak.

No fixtures are committed. Only the maintainer can produce them, and the script
prints what it redacted so the output can be read before `git add`.

ci:local exit 0: 972 root tests (+16), 302 dashboard.
@pitimon
pitimon merged commit 9ffe12f into main Jul 25, 2026
1 check passed
@pitimon
pitimon deleted the feat/105-capture-fixtures branch July 25, 2026 23:58
pitimon added a commit that referenced this pull request Jul 26, 2026
15 commits since v0.39.43. Tracker: #125.

User-facing:
- The Projects panel honours the date range. It read no query parameter at
  all, so picking "24h" narrowed every other card while Projects kept showing
  all-time totals with nothing on screen saying so (#118).
- Per-repo cost, with rows written before the change named as unpriced rather
  than shown as a confident $0 (#121).
- Sources with no per-repo attribution are named, instead of their absence
  reading as "that tool cost nothing here" (#118).
- A plan-value card answering what README:32 already promised, labelled as
  list-price-equivalent rather than a saving (#122).
- The Codex quota chip no longer vanishes on a 401 (#119).
- `sync --compact`, which reclaimed 34,924 lines to 5,636 on a real install
  (#117).

Behind the scenes: outbound-privacy validator hardening (#111, #114), avatar
proxy per-hop revalidation, port-aware allowlisting and a real download cap
(#109, #112), a parser conformance ratchet (#116, #120), scripts/graph out of
the product gate (#115), and a usage-limits fixture capture tool (#124).

Version bumped in all four lockstep locations by the `version` hook:
package.json, package-lock.json, both TokenTrackerBar targets, and the Windows
csproj. validate:version-lockstep passes.

WATCH AFTER SHIPPING: #121 migrates cursors.json on every user's first sync
after upgrading, re-keying project buckets from project|source|hour to
project|source|model|hour. Without that the old bucket strands and its usage is
counted twice. Verified read-only against a real 2,015-bucket state — every key
migrated, total_tokens preserved exactly at 6,281,653,062, and all 2,015
queuedKey markers survived, the loss of which would re-append every row. That
is one machine; a differently-shaped state is the residual risk.

ci:local exit 0: 972 root tests, 302 dashboard.

Co-authored-by: itarun.p <itarun.p@somapait.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant