I’d like to contribute the roadmap item for tiktoken integration to improve token counting accuracy.
Why:
current token counting is approximate
chunk boundaries and chunk sizes can drift from model-actual tokenization
accurate counting should improve chunking consistency and retrieval quality
Proposed scope:
add tiktoken-backed token counting in tokenizer logic
keep current approximation as fallback when tiktoken is unavailable
avoid breaking API changes
update docs for behavior and fallback
Testing:
add regression tests for token counting behavior
include edge cases (empty text, unicode, long text)
run full suite with make installcheck
If this direction looks good, I can open a PR linked to this issue.
I’d like to contribute the roadmap item for tiktoken integration to improve token counting accuracy.
Why:
current token counting is approximate
chunk boundaries and chunk sizes can drift from model-actual tokenization
accurate counting should improve chunking consistency and retrieval quality
Proposed scope:
add tiktoken-backed token counting in tokenizer logic
keep current approximation as fallback when tiktoken is unavailable
avoid breaking API changes
update docs for behavior and fallback
Testing:
add regression tests for token counting behavior
include edge cases (empty text, unicode, long text)
run full suite with make installcheck
If this direction looks good, I can open a PR linked to this issue.