@localmode/transformers

HuggingFace Transformers.js provider for LocalMode — run ML models locally in the browser.

Features

Browser-Native - Run ML models directly in the browser with WebGPU/WASM
Privacy-First - All processing happens locally, no data leaves the device
Model Caching - Model files are cached in the browser Cache API (transformers-cache) for instant subsequent loads, with a resilient write path that never fails a model load (opt out: createTransformers({ resilientCache: false }))
Optimized - Uses quantized models for smaller size and faster inference

Installation

pnpm install @localmode/transformers @localmode/core

Dependencies

Package	Purpose
`@huggingface/transformers` (^4.2.0)	ML inference via ONNX Runtime (WebGPU/WASM)
`phonemizer`	eSpeak-NG WASM for Kokoro TTS text-to-phoneme conversion

Overview

@localmode/transformers provides model implementations for the interfaces defined in @localmode/core. It wraps HuggingFace Transformers.js to enable local ML inference in the browser.

Provider API

All models are created via the transformers provider object. Each factory method returns a model implementing a @localmode/core interface.

Embeddings — Docs

import { embed, embedMany } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const embeddingModel = transformers.embedding('Xenova/bge-small-en-v1.5');

const { embedding } = await embed({ model: embeddingModel, value: 'Hello world' });
const { embeddings } = await embedMany({ model: embeddingModel, values: ['Hello', 'World'] });

Method	Interface	Description
`transformers.embedding(modelId)`	`EmbeddingModel`	Text embeddings

Recommended Models:

Xenova/all-MiniLM-L6-v2 - Fast, general-purpose (~22MB)
Xenova/paraphrase-multilingual-MiniLM-L12-v2 - 50+ languages

Multimodal Embeddings (CLIP/SigLIP) — Docs

Embed both text and images into the same vector space for cross-modal search.

import { embed, embedImage, cosineSimilarity } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const model = transformers.multimodalEmbedding('Xenova/clip-vit-base-patch32');

// Text embedding
const { embedding: textVec } = await embed({ model, value: 'a photo of a cat' });

// Image embedding (same vector space)
const { embedding: imgVec } = await embedImage({ model, image: catImageBlob });

// Cross-modal similarity
const similarity = cosineSimilarity(textVec, imgVec);

Method	Interface	Description
`transformers.multimodalEmbedding(modelId)`	`MultimodalEmbeddingModel`	Text + image embeddings

Recommended Models:

Xenova/clip-vit-base-patch32 - Fast, 512 dimensions
Xenova/clip-vit-base-patch16 - Better accuracy, 512 dimensions

Reranking — Docs

import { rerank } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const rerankerModel = transformers.reranker('Xenova/ms-marco-MiniLM-L-6-v2');

const { results } = await rerank({
  model: rerankerModel,
  query: 'What is machine learning?',
  documents: ['ML is a subset of AI...', 'Python is a language...'],
  topK: 5,
});

Method	Interface	Description
`transformers.reranker(modelId)`	`RerankerModel`	Document reranking

Classification & NLP — Docs

import { classify, extractEntities } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const sentiment = await classify({
  model: transformers.classifier('Xenova/distilbert-base-uncased-finetuned-sst-2-english'),
  text: 'I love this product!',
});

const entities = await extractEntities({
  model: transformers.ner('Xenova/bert-base-NER'),
  text: 'John works at Microsoft in Seattle',
});

Method	Interface	Description
`transformers.classifier(modelId)`	`ClassificationModel`	Text classification
`transformers.zeroShot(modelId)`	`ZeroShotClassificationModel`	Zero-shot text classification
`transformers.ner(modelId)`	`NERModel`	Named Entity Recognition

Translation & Summarization

Method	Interface	Description	Docs
`transformers.translator(modelId)`	`TranslationModel`	Text translation	Docs
`transformers.summarizer(modelId)`	`SummarizationModel`	Text summarization	Docs
`transformers.fillMask(modelId)`	`FillMaskModel`	Masked token prediction	Docs
`transformers.questionAnswering(modelId)`	`QuestionAnsweringModel`	Extractive QA	Docs

Audio

import { transcribe, synthesizeSpeech } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const transcription = await transcribe({
  model: transformers.speechToText('onnx-community/moonshine-tiny-ONNX'),
  audio: audioBlob,
  returnTimestamps: true,
});

const { audio, sampleRate } = await synthesizeSpeech({
  model: transformers.textToSpeech('onnx-community/Kokoro-82M-v1.0-ONNX'),
  text: 'Hello, how are you?',
  voice: 'af_heart', // 29 English voices (see KOKORO_VOICES)
  speed: 1.0,        // 0.5 – 2.0
});

Method	Interface	Description	Docs
`transformers.speechToText(modelId)`	`SpeechToTextModel`	Speech-to-text transcription	Docs
`transformers.textToSpeech(modelId)`	`TextToSpeechModel`	Text-to-speech synthesis	Docs
`transformers.audioClassifier(modelId)`	`AudioClassificationModel`	Audio classification
`transformers.zeroShotAudioClassifier(modelId)`	`ZeroShotAudioClassificationModel`	Zero-shot audio classification
`transformers.vad(modelId)`	`VADProvider`	Voice Activity Detection (Silero)

Voice Activity Detection (VAD)

Detect speech segments in real-time audio streams. Used with createLiveTranscriber() for open-mic and push-to-talk transcription.

import { createLiveTranscriber } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const vad = transformers.vad('onnx-community/silero-vad');
const transcriber = await createLiveTranscriber({
  model: transformers.speechToText('onnx-community/moonshine-tiny-ONNX'),
  mode: 'open-mic',
  vad,
});

Method	Interface	Description
`transformers.vad(modelId)`	`VADProvider`	Voice Activity Detection (Silero VAD)

Recommended Models:

Model	Description
`onnx-community/silero-vad`	Silero VAD v5 — recommended browser VAD (~1.8MB, 512-sample frames at 16 kHz)

Options: threshold (speech probability, default 0.5), silenceTimeoutMs (end-of-utterance timeout, default 700).

Vision

import { classifyImage, captionImage } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const classification = await classifyImage({
  model: transformers.imageClassifier('Xenova/vit-base-patch16-224'),
  image: imageBlob,
});

const caption = await captionImage({
  model: transformers.captioner('onnx-community/Florence-2-base-ft'),
  image: imageBlob,
});

Method	Interface	Description	Docs
`transformers.imageClassifier(modelId)`	`ImageClassificationModel`	Image classification	Docs
`transformers.zeroShotImageClassifier(modelId)`	`ZeroShotImageClassificationModel`	Zero-shot image classification	Docs
`transformers.captioner(modelId)`	`ImageCaptionModel`	Image captioning	Docs
`transformers.segmenter(modelId)`	`SegmentationModel`	Image segmentation	Docs
`transformers.objectDetector(modelId)`	`ObjectDetectionModel`	Object detection	Docs
`transformers.imageFeatures(modelId)`	`ImageFeatureModel`	Image feature extraction	Docs
`transformers.imageToImage(modelId)`	`ImageToImageModel`	Image super resolution	Docs
`transformers.depthEstimator(modelId)`	`DepthEstimationModel`	Monocular depth estimation

OCR & Document QA

Method	Interface	Description	Docs
`transformers.ocr(modelId)`	`OCRModel`	OCR (TrOCR, GLM-OCR, LightOnOCR-2)	Docs
`transformers.documentQA(modelId)`	`DocumentQAModel`	Document/Table question answering	Docs

Text Generation — Docs

Run ONNX-format language models in the browser with WebGPU acceleration:

import { generateText, streamText } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const model = transformers.languageModel('onnx-community/Qwen3.5-0.8B-ONNX');

// Single-shot generation
const { text } = await generateText({ model, prompt: 'What is 2+2?' });

// Streaming generation
const result = await streamText({ model, prompt: 'Write a haiku' });
for await (const chunk of result.stream) {
  process.stdout.write(chunk.text);
}

Method	Interface	Description
`transformers.languageModel(modelId)`	`LanguageModel`	Text generation (ONNX, WebGPU/WASM)

Recommended ONNX LLMs (16 curated models):

Model	Size	Context	Vision
`onnx-community/granite-4.0-350m-ONNX-web`	~120MB	4K	No
`onnx-community/Qwen3-0.6B-ONNX`	~570MB	4K	No
`onnx-community/Qwen3.5-0.8B-ONNX`	~500MB	32K	Yes
`onnx-community/granite-4.0-1b-ONNX-web`	~350MB	4K	No
`onnx-community/Llama-3.2-1B-Instruct-ONNX`	~380MB	8K	No
`onnx-community/TinyLlama-1.1B-Chat-v1.0-ONNX`	~350MB	2K	No
`onnx-community/Qwen2.5-Coder-1.5B-Instruct`	~450MB	4K	No
`onnx-community/DeepSeek-R1-Distill-Qwen-1.5B-ONNX`	~500MB	4K	No
`onnx-community/Llama-3.2-3B-Instruct-ONNX`	~900MB	8K	No
`onnx-community/Qwen3-4B-ONNX`	~1.2GB	4K	No
`microsoft/Phi-3-mini-4k-instruct-onnx-web`	~1.2GB	4K	No
`onnx-community/Qwen3.5-2B-ONNX`	~1.5GB	32K	Yes
`onnx-community/gemma-4-E2B-it-ONNX`	~1.5GB	128K	Yes
`onnx-community/Phi-4-mini-instruct-web-q4f16`	~2.3GB	4K	No
`onnx-community/Qwen3.5-4B-ONNX`	~2.5GB	32K	Yes
`onnx-community/gemma-4-E4B-it-ONNX`	~3GB	128K	Yes

Vision support: Qwen3.5, Qwen2.5-VL, Qwen3-VL, and Gemma 4 models support image input via their built-in vision encoder. Check model.supportsVision for feature detection. See Vision docs for usage.

Model Utilities

import { preloadModel, isModelCached, getModelStorageUsage } from '@localmode/transformers';

const cached = await isModelCached('Xenova/bge-small-en-v1.5');

await preloadModel('Xenova/bge-small-en-v1.5', {
  onProgress: (p) => console.log(`${p.progress}% loaded`),
});

const usage = await getModelStorageUsage();

Resilient Model-File Cache

Model files are cached in the browser Cache API (cache name transformers-cache, honoring a user-set env.cacheKey) — not IndexedDB. The provider installs a resilient wrapper by default so a failing cache write (e.g. the intermittent NetworkError some browsers throw mid-write, or QuotaExceededError) never fails a model load: the model still loads from the network response, and one warning per URL per session is logged. Environments without caches (Node/SSR) skip installation and keep Transformers.js's own file cache.

import { createTransformers } from '@localmode/transformers';

// Opt out of the resilient cache (global — the transformers.js env is a singleton)
const provider = createTransformers({ resilientCache: false });

// Manual control (advanced)
import {
  createResilientModelCache,   // build a Cache-API-backed cache implementing match/put/delete
  installResilientModelCache,  // idempotently configure env.useCustomCache + env.customCache
  setResilientModelCacheEnabled, // enable/disable installation globally
} from '@localmode/transformers';

Recommended Models

Embeddings

Model	Description
`Xenova/bge-small-en-v1.5`	Fast, general-purpose (~22MB, 384d)
`Xenova/paraphrase-multilingual-MiniLM-L12-v2`	50+ languages (~120MB, 384d)
`Xenova/all-mpnet-base-v2`	Higher quality (~420MB, 768d)
`Snowflake/snowflake-arctic-embed-xs`	Tiny retrieval embeddings (~23MB, 384d)

Reranking

Model	Description
`Xenova/ms-marco-MiniLM-L-6-v2`	Fast, small (~23MB, recommended)

Text Classification

Model	Description
`Xenova/distilbert-base-uncased-finetuned-sst-2-english`	Sentiment analysis
`Xenova/twitter-roberta-base-sentiment-latest`	Twitter sentiment

Zero-Shot Classification

Model	Description
`Xenova/mobilebert-uncased-mnli`	Fast, mobile-friendly (~21MB)
`Xenova/nli-deberta-v3-xsmall`	Mid-tier accuracy (~90MB)

Named Entity Recognition

Model	Description
`Xenova/bert-base-NER`	Standard NER (PER, ORG, LOC, MISC)

Translation

Model	Description
`Xenova/opus-mt-en-de`	English to German
`Xenova/opus-mt-en-fr`	English to French
`Xenova/opus-mt-en-es`	English to Spanish

Summarization

Model	Description
`Xenova/distilbart-cnn-6-6`	Best quality browser summarizer (~284MB)

Fill-Mask

Model	Description
`onnx-community/ModernBERT-base-ONNX`	General purpose (mask: `[MASK]`)

Question Answering

Model	Description
`Xenova/distilbert-base-cased-distilled-squad`	SQuAD trained (~65MB)

Speech-to-Text

Model	Description
`onnx-community/moonshine-tiny-ONNX`	Fast, edge-optimized (~50MB)
`onnx-community/moonshine-base-ONNX`	Best quality/size ratio (~237MB)

Text-to-Speech

Model	Description
`onnx-community/Kokoro-82M-v1.0-ONNX`	Natural speech, 29 English voices (~86MB)

Image Classification

Model	Description
`Xenova/vit-base-patch16-224`	General image classification
`Xenova/siglip-base-patch16-224`	Zero-shot image classification (~400MB)

Image Captioning

Model	Description
`onnx-community/Florence-2-base-ft`	High-quality captions (~223MB)

Image Segmentation

Model	Description
`Xenova/segformer-b0-finetuned-ade-512-512`	Semantic segmentation (ADE20K)

Object Detection

Model	Description
`onnx-community/dfine_n_coco-ONNX`	State-of-the-art, tiny (~4.5MB)
`Xenova/detr-resnet-50`	Classic transformer-based detection

Image Features

Model	Description
`Xenova/siglip-base-patch16-224`	Image embeddings (768d)
`onnx-community/dinov2-base-ONNX`	Self-supervised features

Image Super Resolution

Model	Description
`Xenova/swin2SR-lightweight-x2-64`	2x upscale, fast
`Xenova/swin2SR-classical-sr-x4-64`	4x upscale

OCR

Model	Description
`Xenova/trocr-small-printed`	Printed text, line-level (~120MB)
`Xenova/trocr-small-handwritten`	Handwritten text, line-level (~120MB)
`onnx-community/GLM-OCR-ONNX`	Document-level OCR with table/formula recognition (~652MB)
`onnx-community/LightOnOCR-2-1B-ONNX`	Fast document OCR, 11 languages (~700MB)

Document QA

Model	Description
`onnx-community/Florence-2-base-ft`	Document QA (~223MB)
`Xenova/donut-base-finetuned-docvqa`	Donut (~218MB)

Model Constants

All recommended models are exported as constants for easy reference:

import {
  MODELS,                      // All models organized by task
  EMBEDDING_MODELS,
  CLASSIFICATION_MODELS,
  ZERO_SHOT_MODELS,
  NER_MODELS,
  RERANKER_MODELS,
  SPEECH_TO_TEXT_MODELS,
  TEXT_TO_SPEECH_MODELS,
  IMAGE_CLASSIFICATION_MODELS,
  ZERO_SHOT_IMAGE_MODELS,
  IMAGE_CAPTION_MODELS,
  TRANSLATION_MODELS,
  SUMMARIZATION_MODELS,
  FILL_MASK_MODELS,
  QUESTION_ANSWERING_MODELS,
  OBJECT_DETECTION_MODELS,
  SEGMENTATION_MODELS,
  OCR_MODELS,
  DOCUMENT_QA_MODELS,
  IMAGE_TO_IMAGE_MODELS,
  IMAGE_FEATURE_MODELS,
  VAD_MODELS,
  TRANSFORMERS_LLM_MODELS,
  MULTIMODAL_EMBEDDING_MODELS,
  KOKORO_LANG_MAP,
} from '@localmode/transformers';

// Use with provider
const model = transformers.embedding(EMBEDDING_MODELS.BGE_SMALL_EN);

Kokoro Voice Catalog

The KOKORO_VOICES export provides a catalog of 29 English voices with metadata for UI display:

import { KOKORO_VOICES, KOKORO_DEFAULT_VOICE } from '@localmode/transformers';
import type { KokoroVoice } from '@localmode/transformers';

// Each voice has: id, name, language, languageLabel, gender
const english = KOKORO_VOICES.filter((v) => v.language === 'en-US');
const females = KOKORO_VOICES.filter((v) => v.gender === 'female');

console.log(KOKORO_DEFAULT_VOICE); // 'af_heart'

Languages: American English, British English.

Advanced Usage

Custom Model Options

const model = transformers.embedding('Xenova/bge-small-en-v1.5', {
  quantized: true, // Use quantized model (smaller, faster)
  device: 'webgpu', // Use WebGPU for acceleration (falls back to WASM)
});

Language Model Options

Language models accept additional settings via LanguageModelSettings:

const model = transformers.languageModel('onnx-community/Qwen3.5-0.8B-ONNX', {
  contextLength: 32768,
  maxTokens: 1024,
  temperature: 0.7,
  device: 'webgpu',
  // dtype accepts a string or a per-component config object
  dtype: 'q4f16',
  // For multimodal models, use per-component dtype:
  // dtype: { embed_tokens: 'q4', vision_encoder: 'q4', decoder_model_merged: 'q4' },
});

Provider Options

Pass provider-specific options to core functions:

const { embedding } = await embed({
  model: transformers.embedding('Xenova/bge-small-en-v1.5'),
  value: 'Hello world',
  providerOptions: {
    transformers: {
      // Any Transformers.js specific options
    },
  },
});

Preloading Models

For better UX, preload models before use:

import { preloadModel, isModelCached } from '@localmode/transformers';
import { embed } from '@localmode/core';

if (!(await isModelCached('Xenova/bge-small-en-v1.5'))) {
  await preloadModel('Xenova/bge-small-en-v1.5', {
    onProgress: (p) => console.log(`Loading: ${p.progress}%`),
  });
}

// Subsequent calls are instant (loaded from cache)
const embeddingModel = transformers.embedding('Xenova/bge-small-en-v1.5');
const { embedding } = await embed({ model: embeddingModel, value: 'Hello' });

Exported Implementation Classes

For advanced use cases, implementation classes are available:

import {
  TransformersEmbeddingModel,
  TransformersClassificationModel,
  TransformersZeroShotModel,
  TransformersNERModel,
  TransformersRerankerModel,
  TransformersSpeechToTextModel,
  TransformersImageClassificationModel,
  TransformersZeroShotImageModel,
  TransformersCaptionModel,
  TransformersCLIPEmbeddingModel,
  TransformersLanguageModel,
  TransformersGenerativeOCRModel,
  isGenerativeOCRModel,
  TransformersSileroVAD,
  createSileroVAD,
} from '@localmode/transformers';

Browser Compatibility

Browser	WebGPU	WASM	Notes
Chrome 113+	✅	✅	Best performance with WebGPU
Edge 113+	✅	✅	Same as Chrome
Firefox	❌	✅	WASM only
Safari 26+	✅	✅	WebGPU available
iOS Safari	✅	✅	WebGPU available (iOS 26+)

Performance Tips

Use quantized models - Smaller and faster with minimal quality loss
Preload models - Load during app init for instant inference
Use WebGPU when available - 3-5x faster than WASM
Batch operations - Process multiple inputs together

Acknowledgments

This package is built on Transformers.js by HuggingFace — state-of-the-art ML models running in the browser via ONNX Runtime.

License

MIT

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

@localmode/transformers

Features

Installation

Dependencies

Overview

Provider API

Embeddings — Docs

Multimodal Embeddings (CLIP/SigLIP) — Docs

Reranking — Docs

Classification & NLP — Docs

Translation & Summarization

Audio

Voice Activity Detection (VAD)

Vision

OCR & Document QA

Text Generation — Docs

Model Utilities

Resilient Model-File Cache

Recommended Models

Embeddings

Reranking

Text Classification

Zero-Shot Classification

Named Entity Recognition

Translation

Summarization

Fill-Mask

Question Answering

Speech-to-Text

Text-to-Speech

Image Classification

Image Captioning

Image Segmentation

Object Detection

Image Features

Image Super Resolution

OCR

Document QA

Model Constants

Kokoro Voice Catalog

Advanced Usage

Custom Model Options

Language Model Options

Provider Options

Preloading Models

Exported Implementation Classes

Browser Compatibility

Performance Tips

Acknowledgments

License