feat(node): expose the pipeline encode path over napi

[ArthurZucker]

Tokenizer.encode goes through encode_char_offsets, so none of the pipeline work is reachable from Node. Two entry points, ids only, no offsets.

The boundary is the whole story once the encode is fast, so the shapes are picked around it: encode returns a Uint32Array because a Vec<u32> marshals as a boxed JS Array (one napi value per token — 13× the encode itself on token-dense input), and encodeBytesInto drops the two remaining per-call costs, the JS string → UTF-8 copy and the fresh ArrayBuffer.

import { PipelineTokenizer } from 'tokenizers'
const p = PipelineTokenizer.fromFile('tokenizer.json')
p.encode('Hello World')                    // Uint32Array
p.encodeBytesInto(buf, out)                 // n ids written into `out`

Bench

// Boundary cost of the two shipped bindings, against the pure-Rust ceiling
// (tk-encode/examples/node_ref.rs) and @huggingface/tokenizers 0.1.3.
import { readFileSync } from 'node:fs'
import { createRequire } from 'node:module'
import { Tokenizer as Js } from '@huggingface/tokenizers'
const napi = createRequire(import.meta.url)('./bindings/node/index.js')

const C = 'tokenizers/data/corpora'
const SHORT = 'Summarize the following article in three bullet points, then rate its clarity from 1 to 10.'

const mbps = (fn, mb) => {
  fn()
  let b = 0, t = 0
  while (t < 1.5) {
    const s0 = process.hrtime.bigint()
    fn()
    const s = Number(process.hrtime.bigint() - s0) / 1e9
    t += s
    b = Math.max(b, mb / s)
  }
  return b
}
const ns = (fn) => {
  for (let i = 0; i < 2000; i++) fn()
  let b = Infinity
  for (let r = 0; r < 5; r++) {
    const t = process.hrtime.bigint()
    for (let i = 0; i < 50000; i++) fn()
    b = Math.min(b, Number(process.hrtime.bigint() - t) / 50000)
  }
  return b
}

for (const model of process.argv.slice(2)) {
  const d = `models/${model}`
  const p = napi.PipelineTokenizer.fromFile(`${d}/tokenizer.json`)
  const js = new Js(
    JSON.parse(readFileSync(`${d}/tokenizer.json`, 'utf8')),
    JSON.parse(readFileSync(`${d}/tokenizer_config.json`, 'utf8')),
  )
  for (const f of ['english', 'chinese']) {
    const text = readFileSync(`${C}/${f}.txt`, 'utf8')
    const buf = Buffer.from(text, 'utf8')
    const mb = buf.length / 1e6
    const out = new Uint32Array(p.encode(text, true).length + 16)
    console.log(
      `${model.padEnd(14)} ${f.padEnd(8)} encode ${mbps(() => p.encode(text, true), mb).toFixed(0).padStart(5)}` +
      `  encodeBytesInto ${mbps(() => p.encodeBytesInto(buf, out, true), mb).toFixed(0).padStart(5)}` +
      `  js ${mbps(() => js.encode(text), mb).toFixed(1).padStart(5)} MB/s`,
    )
  }
  const sbuf = Buffer.from(SHORT, 'utf8')
  const sout = new Uint32Array(64)
  console.log(
    `${model.padEnd(14)} 89 B     encode ${ns(() => p.encode(SHORT, true)).toFixed(0).padStart(5)}` +
    `  encodeBytesInto ${ns(() => p.encodeBytesInto(sbuf, sout, true)).toFixed(0).padStart(5)}` +
    `  js ${ns(() => js.encode(SHORT)).toFixed(0).padStart(5)} ns/call\n`,
  )
}

M4, single thread, 195 kB corpora, warm cache, ids verified equal to Tokenizer.encode on every cell:

gpt2           english  encode   649  encodeBytesInto  1247  js   8.0 MB/s
gpt2           chinese  encode   528  encodeBytesInto   660  js   9.4 MB/s
gpt2           89 B     encode   777  encodeBytesInto   379  js 12335 ns/call

Qwen2.5-0.5B   english  encode   424  encodeBytesInto   697  js   7.2 MB/s
Qwen2.5-0.5B   chinese  encode   295  encodeBytesInto   313  js  12.6 MB/s
Qwen2.5-0.5B   89 B     encode   850  encodeBytesInto   465  js 15705 ns/call

DeepSeek-V3    english  encode   369  encodeBytesInto   519  js   7.0 MB/s
DeepSeek-V3    chinese  encode   465  encodeBytesInto   564  js  10.5 MB/s
DeepSeek-V3    89 B     encode   922  encodeBytesInto   514  js 17451 ns/call

bert-base-uncased english  encode    60  encodeBytesInto    63  js   7.1 MB/s
bert-base-uncased chinese  encode    47  encodeBytesInto    47  js   6.8 MB/s
bert-base-uncased 89 B     encode  2382  encodeBytesInto  1960  js 11165 ns/call

Boundary legs, gpt2 / 89 B: napi call 49 ns, JS string → UTF-8 109 ns, encode 250 ns, fresh Uint32Array 388 ns, Vec<u32> → JS Array 15 µs. encodeBytesInto lands at 379 ns against a 217 ns pure-Rust ceiling; the remaining ~110 ns is argument marshalling, which only batching removes.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.