Skip to content

Token & cost estimation adapters #13

Description

@Cryptoteep

Context

RunResult reports durationMs but nothing about tokens or cost. For eval suites run in CI, you usually want to know:

  • how many tokens were sent/received,
  • estimated cost per case and per suite,
  • totals across a run.

Proposal

Introduce a token-counting adapter that runPrompt/evalPrompt can call optionally:

import { tiktokenCounter } from 'promptkit/tokens'

const result = await runPrompt(
  summarize,
  { text: '…' },
  { call, tokens: tiktokenCounter('gpt-4o-mini') },
)
// result.tokens: { prompt: 42, completion: 18, total: 60 }
// result.cost: { prompt: 0.000063, completion: 0.000027, total: 0.00009 }

Implementation notes

  • Define a TokenCounter interface: (text: string, role: 'prompt' | 'completion') => number.
  • Define a Pricing map per model, loaded from a JSON file so it can be updated without a release.
  • Ship a tiktoken adapter behind a subpath export promptkit/tokens (tiktoken-wasm is heavy; keep it optional / peer dep).
  • Extend RunResult and EvalResult with optional tokens and cost fields.
  • Add a --cost flag to promptkit eval that prints a cost summary.

Acceptance

  • TokenCounter interface + tiktokenCounter adapter.
  • Pricing JSON + cost computation.
  • RunResult.tokens / RunResult.cost populated when a counter is supplied.
  • CLI --cost summary.
  • Tests + docs.

This is a larger issue — feel free to break it into multiple PRs (interface first, then tiktoken adapter, then pricing).

/label "help wanted" "enhancement"

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions