README.md

@localmode/react

64 React hooks for local-first AI. Embed, chat, classify, transcribe, and more — with built-in loading states, error handling, and cancellation.

Installation

pnpm install @localmode/react @localmode/core

Quick Start

import { useChat } from '@localmode/react';
import { webllm } from '@localmode/webllm';

const model = webllm.languageModel('Llama-3.2-1B-Instruct-q4f16_1-MLC');

function Chat() {
  const { messages, isStreaming, send, cancel } = useChat({ model });

  return (
    <div>
      {messages.map((m) => (
        <div key={m.id}><b>{m.role}:</b> {m.content}</div>
      ))}
      <button onClick={() => send('What is LocalMode?')}>Send</button>
      {isStreaming && <button onClick={cancel}>Stop</button>}
    </div>
  );
}

Domain Hooks

Hook	Domain	Wraps
useEmbed	Embeddings	embed()
useEmbedMany	Embeddings	embedMany()
useSemanticSearch	Embeddings	semanticSearch()
useEmbedImage	Multimodal	embedImage() — CLIP cross-modal
useEmbedManyImages	Multimodal	streamEmbedManyImages() — batch image embedding with progress
useChat	Generation	streamText() with message state, vision images, usage tracking (usage/totalUsage), lifecycle status, and regenerate() reply variants
useGenerateText	Generation	generateText()
useGenerateObject	Generation	generateObject() — typed JSON output
useClassify	Classification	classify()
useClassifyZeroShot	Classification	classifyZeroShot()
useExtractEntities	NER	extractEntities()
useRerank	Classification	rerank() — reorder documents by relevance to a query, with hook-level/per-call topK
useTranscribe	Audio	transcribe()
useSynthesizeSpeech	Audio	synthesizeSpeech()
useStreamSpeech	Audio	streamSynthesizeSpeech() — clause-by-clause streaming TTS
useCaptionImage	Vision	captionImage()
useDetectObjects	Vision	detectObjects()
useClassifyImage	Vision	classifyImage()
useClassifyImageZeroShot	Vision	classifyImageZeroShot()
useSegmentImage	Vision	segmentImage()
useExtractImageFeatures	Vision	extractImageFeatures()
useImageToImage	Vision	imageToImage()
useDetectHands	Vision	detectHands()
useDetectPose	Vision	detectPose()
useDetectFace	Vision	detectFace()
useDetectFaceLandmarks	Vision	detectFaceLandmarks()
useRecognizeGesture	Vision	recognizeGesture()
useDetectLanguage	Text	detectLanguage()
useTranslate	Text	translate()
useSummarize	Text	summarize()
useExtractText	OCR	extractText()
useFillMask	NLP	fillMask()
useAnswerQuestion	QA	answerQuestion()
useAskDocument	Document QA	askDocument()
useAgent	Agents	createAgent() + runAgent() — ReAct loop with tools; human-in-the-loop approval for requiresApproval tools via pendingApproval + approve()/deny(reason?)
useImportExport	Import/Export	importFrom(), exportToCSV(), exportToJSONL()
useEvaluateModel	Evaluation	evaluateModel() — run metrics against a dataset
useSemanticChunk	RAG	semanticChunk() — embedding-aware topic-boundary chunking
useCalibrateThreshold	Embeddings	calibrateThreshold() — empirical similarity threshold
useAuditLog	Security	createAuditLog() — tamper-evident hash-chained log
useEncryptedVault	Security	deriveEncryptionKey() + AES-GCM envelopes over a pluggable StorageAdapter — passphrase-locked encrypted CRUD vault
useLiveTranscribe	Audio	createLiveTranscriber() — streaming mic STT with VAD
useTurnTaker	Audio	createTurnTaker() — full voice loop orchestrator
useStreamingTracker	Vision	provider streaming trackers (e.g. mediapipe.createHandTracker()) — real-time video tracking with latest results + fps (experimental)
useProviderFallback	Generation	per-capability Chrome Built-in AI ⇄ Transformers.js resolution (resolveSummarizer/resolveTranslator/resolveEditEngine/resolveFillMask) — providers loaded via dynamic import() (no hard provider dep), session-cached, with truthful resolution provenance for a badge. Chrome AI is selected only when availability() === 'available'; chromeAvailability/refreshChromeAvailability/requestChromeDownload/chromeDownloadProgress/downloadingCapability drive a download gate, since Chrome needs a user activation (a click) to fetch its one-time, browser-wide model
usePhotoLibrary	Multimodal	shared in-memory CLIP-family photo library — one injected model powers both multimodal embeddings and zero-shot categorization; progressive/adaptive/cancellable ingest, text+image search over one vector space, union-find dedup, confirmed model-switch re-index
useKnowledgeBase	RAG	knowledge-base session orchestration over the core KnowledgeBaseEngine contract — raw-document store, useModelLoad lifecycle, chunking config, and re-ingest on engine-kind toggle OR embedding-model switch via an injected createEngine(kind) factory (langchain path optional/dynamic at the block layer)

Utility Hooks

Hook	Purpose
useModelLoad	Provider model load lifecycle — singleton model creation, normalized cross-provider download progress (0–1), warmup-driven status
useModelStatus	Real model readiness (isReady/isLoading/progress/error) from the useModelLoad registry
useCapabilities	Detect browser AI capabilities (typed DeviceCapabilities, with refresh())
useNetworkStatus	Online/offline status
useStorageQuota	Storage quota monitoring (full StorageQuota incl. isPersisted/availableBytes)
useVoiceRecorder	MediaRecorder lifecycle with mic selection (deviceId/constraints), live stream, and getVolume()
useInferenceQueue	Priority-based task scheduling with live stats
useSemanticCache	Semantic cache lifecycle (create/destroy, stats)
useReindex	Embedding drift re-embedding with progress and cancellation
useModelRecommendations	Ranked model recommendations by device capabilities
useModelLoader	Chunked model download with LRU eviction and cross-tab coordination
useAdaptiveBatchSize	Device-aware optimal batch size for embeddings/inference

Batch & List Processing

Hook	Purpose
useBatchOperation	Concurrent batch with progress
useOperationList	Accumulate results into a list with item removal
useSequentialBatch	Sequential processing with progress
usePipeline	Multi-step workflows with progress

Helper Utilities

Utility	Purpose
readFileAsDataUrl(file)	Read a File as a data URL string
validateFile({ file, accept, maxSize })	Validate file type/size, returns AppError | null
downloadBlob(content, filename, mimeType?)	Trigger file download from in-memory content
toAppError(error, recoverable?)	Convert Error to AppError shape (carries over LocalModeError code, appends hint to the message)
useObjectUrl(blob)	Hook: object URL for a Blob with automatic revocation on change/unmount

Encrypted Vault

useEncryptedVault provides a passphrase-locked, encrypted item store persisted through any core StorageAdapter (default: a dedicated IndexedDBStorage). status is 'uninitialized' | 'locked' | 'unlocked'; a single unlock(passphrase) initializes the vault on first use and verifies the passphrase afterwards (wrong passphrase → typed VaultPassphraseError, detected via an encrypted verifier — works even on an empty vault). While unlocked, createItem/readItem/updateItem/deleteItem/refresh round-trip AES-GCM envelopes; while locked they resolve null/false with a VaultLockedError and never touch storage.

import { useEncryptedVault } from '@localmode/react';

function Notes() {
  const { status, items, error, unlock, lock, createItem, deleteItem } =
    useEncryptedVault<{ note: string }>({ name: 'notes' });

  if (status !== 'unlocked') {
    return (
      <button onClick={() => unlock(prompt('Passphrase') ?? '')}>
        {status === 'uninitialized' ? 'Create vault' : 'Unlock'}
      </button>
    );
  }
  return (
    <div>
      {items.map((i) => (
        <div key={i.id}>
          {i.data.note} <button onClick={() => deleteItem(i.id)}>x</button>
        </div>
      ))}
      <button onClick={() => createItem({ note: 'hello' })}>Add</button>
      <button onClick={lock}>Lock</button>
      {error?.name === 'VaultPassphraseError' && <p>Wrong passphrase</p>}
    </div>
  );
}

Security notes: the key is derived once per unlock via core deriveEncryptionKey (PBKDF2, configurable iterations) and the non-extractable CryptoKey is held in memory only — the passphrase is never retained, and nothing derived from it is persisted except a random salt and the AES-GCM verifier ciphertext. lock() and unmount clear the key and all decrypted state. At rest, everything except item ids and timestamps is ciphertext (fresh 12-byte IV per write, versioned envelopes).

Features

• 47 domain hooks — One for each AI capability in @localmode/core (including reranking, agents, import/export, evaluation, semantic chunking, threshold calibration, MediaPipe landmarks/gestures, real-time video tracking, language detection, audit log, encrypted vault, live transcription, streaming speech, provider fallback, shared photo library, knowledge-base orchestration)
• 12 utility hooks — Model load lifecycle, model status, capabilities, network, storage, voice recording, inference queue, semantic cache, reindex, model recommendations, model loader, adaptive batch size
• 4 batch/pipeline hooks — List accumulation, concurrent batch, sequential batch, pipeline (batch hooks publish results incrementally as items complete)
• 10 pipeline step factories — embedStep, chunkStep, semanticChunkStep, searchStep, rerankStep, storeStep, classifyStep, summarizeStep, generateStep, embedManyStep
• 5 helper utilities — File reading, validation, download, error conversion, object URLs (useObjectUrl)
• Zero dependencies — only peer deps on react and @localmode/core
• Streaming — useChat with real-time message updates, IndexedDB persistence, per-turn + cumulative usage, lifecycle status, and regenerate() reply variants
• Cancellation — every inference hook supports AbortSignal-based cancellation (provider model loads in useModelLoad are not abortable)
• SSR-safe — no-op during server rendering for Next.js compatibility
• Provider-agnostic — works with any @localmode provider

Documentation

localmode.dev/docs/react

License

MIT