Skip to content

feat(transform): adaptive chars-per-token for the profitability gate (opt-in)#128

Open
dhruvraajeev wants to merge 1 commit into
teamchong:mainfrom
dhruvraajeev:feat/adaptive-cpt
Open

feat(transform): adaptive chars-per-token for the profitability gate (opt-in)#128
dhruvraajeev wants to merge 1 commit into
teamchong:mainfrom
dhruvraajeev:feat/adaptive-cpt

Conversation

@dhruvraajeev

Copy link
Copy Markdown

The gate weighs image tokens (exact, from the 28px patch grid) against text tokens (chars / CHARS_PER_TOKEN). That CPT constant is hand-tuned: 4 for reminders/tool_results, 2.0 for slab/history. docs/ADAPTIVE_CPT_PLAN.md measured real density near ~1.5 on dense content, so the gate over-estimates how cheap text is and declines compressions that would have won.

This learns CPT from telemetry pxpipe already writes. bucket_chars, baseline_tokens and image_pixels have been logged for a while and nothing read them; a 6-bucket least-squares fit over those rows solves a marginal token rate per bucket (CPT = 1/slope). No new dependencies.

src/core/cpt-fit.ts the fit. Pure math, no fs, safe in the Workers build.
src/cpt-store.ts Node-only. Reads events.jsonl, writes cpt-state.jsonl,
hands the gate a resolver.

Fails closed everywhere, which is the part worth reviewing. Min 20 samples, min 8 bucket appearances, plausible band 0.8-6.0, condition-number check, non-positive slope, throwing resolver — each rejects that one bucket back to its baked constant and records why in cpt-state.jsonl. A wrong learned rate costs money; a missing one is just today's behavior. Precedence is explicit host charsPerToken -> learned -> constant. Workers keeps the constants outright.

Image cost is subtracted on the patch grid from anthropic-vision.ts, not the retired pixels/750 figure. Pages are 1568x728 and both edges are multiples of 28, so pixels/784 reproduces the patch count exactly (56x26 = 1456).

Off unless PXPIPE_ADAPTIVE_CPT=1; refits are throttled and never block a request. Verified: the fit recovers CPTs it was generated from within 5% including under noise, and a learned rate flips a real gate verdict both ways. Savings on production traffic are NOT measured — that needs a shadow-mode A/B, which is why this is opt-in. Reproduce: npx tsx eval/adaptive-cpt/demo-cpt.mts

…(opt-in)

The gate weighs image tokens (exact, from the 28px patch grid) against text
tokens (chars / CHARS_PER_TOKEN). That CPT constant is hand-tuned: 4 for
reminders/tool_results, 2.0 for slab/history. docs/ADAPTIVE_CPT_PLAN.md measured
real density near ~1.5 on dense content, so the gate over-estimates how cheap
text is and declines compressions that would have won.

This learns CPT from telemetry pxpipe already writes. bucket_chars,
baseline_tokens and image_pixels have been logged for a while and nothing read
them; a 6-bucket least-squares fit over those rows solves a marginal token rate
per bucket (CPT = 1/slope). No new dependencies.

  src/core/cpt-fit.ts   the fit. Pure math, no fs, safe in the Workers build.
  src/cpt-store.ts      Node-only. Reads events.jsonl, writes cpt-state.jsonl,
                        hands the gate a resolver.

Fails closed everywhere, which is the part worth reviewing. Min 20 samples, min
8 bucket appearances, plausible band 0.8-6.0, condition-number check,
non-positive slope, throwing resolver — each rejects that one bucket back to its
baked constant and records why in cpt-state.jsonl. A wrong learned rate costs
money; a missing one is just today's behavior. Precedence is explicit host
charsPerToken -> learned -> constant. Workers keeps the constants outright.

Image cost is subtracted on the patch grid from anthropic-vision.ts, not the
retired pixels/750 figure. Pages are 1568x728 and both edges are multiples of
28, so pixels/784 reproduces the patch count exactly (56x26 = 1456).

Off unless PXPIPE_ADAPTIVE_CPT=1; refits are throttled and never block a
request. Verified: the fit recovers CPTs it was generated from within 5%
including under noise, and a learned rate flips a real gate verdict both ways.
Savings on production traffic are NOT measured — that needs a shadow-mode A/B,
which is why this is opt-in. Reproduce: npx tsx eval/adaptive-cpt/demo-cpt.mts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant