Context
RunResult reports durationMs but nothing about tokens or cost. For eval suites run in CI, you usually want to know:
- how many tokens were sent/received,
- estimated cost per case and per suite,
- totals across a run.
Proposal
Introduce a token-counting adapter that runPrompt/evalPrompt can call optionally:
import { tiktokenCounter } from 'promptkit/tokens'
const result = await runPrompt(
summarize,
{ text: '…' },
{ call, tokens: tiktokenCounter('gpt-4o-mini') },
)
// result.tokens: { prompt: 42, completion: 18, total: 60 }
// result.cost: { prompt: 0.000063, completion: 0.000027, total: 0.00009 }
Implementation notes
- Define a
TokenCounter interface: (text: string, role: 'prompt' | 'completion') => number.
- Define a
Pricing map per model, loaded from a JSON file so it can be updated without a release.
- Ship a
tiktoken adapter behind a subpath export promptkit/tokens (tiktoken-wasm is heavy; keep it optional / peer dep).
- Extend
RunResult and EvalResult with optional tokens and cost fields.
- Add a
--cost flag to promptkit eval that prints a cost summary.
Acceptance
This is a larger issue — feel free to break it into multiple PRs (interface first, then tiktoken adapter, then pricing).
/label "help wanted" "enhancement"
Context
RunResultreportsdurationMsbut nothing about tokens or cost. For eval suites run in CI, you usually want to know:Proposal
Introduce a token-counting adapter that
runPrompt/evalPromptcan call optionally:Implementation notes
TokenCounterinterface:(text: string, role: 'prompt' | 'completion') => number.Pricingmap per model, loaded from a JSON file so it can be updated without a release.tiktokenadapter behind a subpath exportpromptkit/tokens(tiktoken-wasm is heavy; keep it optional / peer dep).RunResultandEvalResultwith optionaltokensandcostfields.--costflag topromptkit evalthat prints a cost summary.Acceptance
TokenCounterinterface +tiktokenCounteradapter.RunResult.tokens/RunResult.costpopulated when a counter is supplied.--costsummary.This is a larger issue — feel free to break it into multiple PRs (interface first, then tiktoken adapter, then pricing).
/label "help wanted" "enhancement"