Following up on the detailed feedback in my previous PR #3 , I completely agree with the assessment: hand-rolling a regex for BPE fails on dense camelCase identifiers, and undercounting is definitely the dangerous direction for the context window budget.
I'd like to take up your constructive suggestion to vendor an actual zero-dep tokenizer like gpt-tokenizer (pure-JS, no native bindings). However, before I write any code, I want to ensure I strictly align with the CONTRIBUTING.md invariants—specifically:
The Near-zero-deps rule (ts-morph is the exact one runtime dependency).
The Style rule (The published artifact is the single agentmap.mjs file, with no build step).
Since running npm install gpt-tokenizer and adding it to package.json violates the first rule, what is your preferred architectural approach for "vendoring" it?
Option A (In-file copy-paste): Literally copy-paste the core gpt-tokenizer logic directly into the bottom of agentmap.mjs? (Maintains the single-file artifact and zero-build-step rules, but adds significant line count to the file).
Option B (Vendor directory): Place the pure-JS gpt-tokenizer files into a vendor/ directory and import them? (Cleanest repo structure, but breaks the single-file published artifact rule unless a bundler is introduced, which contradicts the "no build step" rule).
Option C: Is there another structure you prefer?
Following up on the detailed feedback in my previous PR #3 , I completely agree with the assessment: hand-rolling a regex for BPE fails on dense
camelCaseidentifiers, and undercounting is definitely the dangerous direction for the context window budget.I'd like to take up your constructive suggestion to vendor an actual zero-dep tokenizer like
gpt-tokenizer(pure-JS, no native bindings). However, before I write any code, I want to ensure I strictly align with theCONTRIBUTING.mdinvariants—specifically:The Near-zero-deps rule (ts-morph is the exact one runtime dependency).
The Style rule (The published artifact is the single
agentmap.mjsfile, with no build step).Since running npm install gpt-tokenizer and adding it to package.json violates the first rule, what is your preferred architectural approach for "vendoring" it?
Option A (In-file copy-paste): Literally copy-paste the core
gpt-tokenizerlogic directly into the bottom of agentmap.mjs? (Maintains the single-file artifact and zero-build-step rules, but adds significant line count to the file).Option B (Vendor directory): Place the pure-JS
gpt-tokenizerfiles into a vendor/ directory and import them? (Cleanest repo structure, but breaks the single-file published artifact rule unless a bundler is introduced, which contradicts the "no build step" rule).Option C: Is there another structure you prefer?