diff --git a/changelog.d/20260712_050900_qwen_wasm_memory.md b/changelog.d/20260712_050900_qwen_wasm_memory.md new file mode 100644 index 0000000..aa8bc35 --- /dev/null +++ b/changelog.d/20260712_050900_qwen_wasm_memory.md @@ -0,0 +1,5 @@ +### Fixed + +- Prevent large Qwen and other ONNX models from being offered on devices whose + WASM heap cannot safely construct the inference session, and guard WebGPU's + CPU fallback with the same device-fit calculation. diff --git a/docs/case-studies/issue-17/README.md b/docs/case-studies/issue-17/README.md new file mode 100644 index 0000000..28d36e9 --- /dev/null +++ b/docs/case-studies/issue-17/README.md @@ -0,0 +1,148 @@ +# Case study — issue #17: Qwen 2.5 instruct loading failed + +## Executive summary + +The screenshot is a Safari session on the deployed GitHub Pages app. SmolLM2 +loads and answers two prompts, proving that the page, worker, ORT WASM backend, +model download, chat template, and generation loop are operational. The user +then selects **Qwen2.5 1.5B Instruct**, shown as **q4f16**, **1.1 GB download**, +**~1.5 GB estimated memory**, and **CPU**. Loading ends with Emscripten's opaque +`Error: Aborted(). Build with -sASSERTIONS for more info.` + +The root cause is the selector's under-estimation of peak memory on the ORT +WASM path. It used `download bytes × 1.3` and compared that with a 2 GiB budget, +but did not include the additional WASM heap needed by ORT's graph, +initializers, allocator, and session construction. That made a 1.22 GB Qwen +graph appear runnable near wasm32's practical limit. The process abort occurs +below JavaScript's recoverable exception boundary, hence the unhelpful error. + +The fix uses a conservative additional 1.8× WASM-heap factor (2.34× download +bytes including the existing model/KV estimate), while retaining the existing +1.3× estimate for WebGPU. Qwen2.5 1.5B is consequently hidden as too large on +CPU-only devices and remains available at q4f16 on WebGPU devices. A failed +WebGPU load is allowed to retry on WASM only when the same model and dtype pass +the CPU fit calculation. The smallest fallback variant is now selected by its +actual byte count, because Qwen's q4 artifact is larger than q8. + +## Preserved evidence + +- [`issue.md`](issue.md) — issue metadata and report summary. +- [`source-assets/issue-17-screenshot.png`](source-assets/issue-17-screenshot.png) + — authenticated download of the original 1824×1115 GitHub attachment. +- GitHub issue comments were queried with pagination: none existed. +- PR #18 conversation comments, inline review comments, and reviews were all + queried separately with pagination: none existed at investigation time. + +## Reconstructed timeline + +| Sequence | Evidence / event | +|---|---| +| 1 | Issue #11 introduced dynamic, device-aware model selection and estimated Transformers.js runtime memory as ONNX download size × 1.3. | +| 2 | Issue #13 fixed ORT asset resolution under the GitHub Pages sub-path. | +| 3 | Issue #15 made the chat surface persist beside the selector. | +| 4 | In the captured Safari session, SmolLM2 135M loaded and generated two coherent replies. This rules out those earlier backend-path and UI defects. | +| 5 | The user selected Qwen2.5 1.5B q4f16. The card classified its estimated ~1.5 GB footprint as fitting the CPU budget. | +| 6 | ORT attempted to construct the much larger session inside its WASM heap and terminated with `Aborted()`. | +| 7 | Issue #17 was filed on 2026-06-12 with the screenshot. | +| 8 | On 2026-07-12 the model tree, selector math, backend dispatch, upstream documentation, and related reports were investigated and the regression tests/fix were added. | + +## Requirements and disposition + +| Requirement | Disposition | +|---|---| +| Fix Qwen2.5 instruct loading failure | Fixed at the cause: CPU-only devices no longer offer Qwen2.5 1.5B when peak WASM memory is unsafe; capable WebGPU devices retain it. | +| Archive all issue evidence | Original attachment and issue metadata are stored in this directory. There were no issue comments or PR review artifacts to archive. | +| Reconstruct sequence and enumerate requirements | This document provides both. | +| Find root causes for every problem | Peak-memory under-estimation is the primary cause; unsafe WebGPU fallback and label-based size fallback were two code-wide variants of it and are also fixed. | +| Research online facts and existing components | Transformers.js, ONNX Runtime Web, the Hub model artifacts, WebLLM, and upstream failure reports were reviewed below. | +| Add diagnostics if root cause remains unknown | Not needed for the selector defect; existing `?debug=1` worker logging remains available and off by default. The improved fallback error is actionable rather than `Aborted()`. | +| Report an upstream issue when another project is at fault | No new upstream issue: the app incorrectly admitted an unsafe configuration. Existing upstream reports already document opaque large-model/OOM failures. | +| Apply the fix everywhere | The shared fit evaluator drives visibility, recommendation, auto-load, manual selection, and WebGPU fallback eligibility. | +| Reproduce with an automated test | Unit tests reproduce the incorrect CPU verdict for Qwen2.5 1.5B and verify its WebGPU verdict. | + +## Root-cause analysis + +### Why this is not a general Qwen or backend failure + +The screenshot shows successful inference immediately before model switching. +The Qwen repository is tagged for Transformers.js and contains the expected +ONNX variants. Transformers.js explicitly demonstrates Qwen2.5 with its +text-generation pipeline. The failure is correlated with the 1.22 GB graph and +CPU/WASM execution, not architecture dispatch or prompting. + +### Faulty memory model + +Before the fix: + +```text +runtime estimate = ONNX download bytes × 1.3 +Qwen q4f16 = 1,221,878,940 × 1.3 ≈ 1.48 GiB +WASM budget = 2.00 GiB +verdict = fits/tight → load allowed +``` + +That estimate covers weights, activations and KV cache but treats the download +almost as the complete live allocation. ORT must deserialize the graph and +allocate session/initializer/arena memory inside the same WASM address space. +The corrected CPU estimate adds a conservative heap construction factor: + +```text +WASM runtime estimate = download × 1.3 × 1.8 ≈ 2.66 GiB +verdict = too large → no unsafe load +``` + +WebGPU retains the original factor because its main tensor allocations are GPU +buffers and do not compete solely inside the wasm32 heap. + +### Unsafe fallback + +The worker previously retried every failed WebGPU pipeline on WASM using the +same model/dtype. A model selected precisely because it fits the larger GPU +budget can exceed the CPU heap, turning a recoverable WebGPU error into an OOM +abort. The main thread now computes the exact CPU verdict and passes fallback +eligibility with the load request. + +### Incorrect “smallest” fallback + +The old fallback assumed `q4f16 < q4 < q8` by label. Actual repositories do not +guarantee that ordering: Qwen2.5 1.5B publishes q4f16 at 1,221,878,940 bytes, +q8 at 1,578,954,293 bytes, and q4 at 1,787,566,590 bytes. The fallback now sorts +available artifacts by measured bytes. + +## Online research and alternatives + +- [Transformers.js documentation](https://huggingface.co/docs/transformers.js/main/index) + identifies WASM as the browser CPU backend, WebGPU as the accelerated path, + and documents q8 as its typical WASM default. It also demonstrates quantized + pipeline selection. +- [Transformers.js v3 announcement](https://huggingface.co/blog/transformersjs-v3) + demonstrates `onnx-community/Qwen2.5-0.5B-Instruct` with q4 on WebGPU and + explains that available dtypes are model-dependent. +- [ONNX Runtime Web support matrix](https://onnxruntime.ai/docs/get-started/with-javascript/web.html) + shows WASM across Safari while WebGPU support is browser-dependent. This + matches the screenshot's Safari + CPU combination. +- [Transformers.js issue #1024](https://github.com/huggingface/transformers.js/issues/1024) + records opaque numeric/load failures for larger text-generation models; an + upstream maintainer identifies out-of-memory as a likely cause. This is + consistent evidence, not proof specific to this session. +- [Qwen2.5 1.5B ONNX repository](https://huggingface.co/onnx-community/Qwen2.5-1.5B-Instruct/tree/main/onnx) + is the authoritative artifact source. Its real sizes demonstrate why dtype + labels cannot serve as a size ordering. +- **WebLLM/MLC** was considered as an alternative engine. It is well optimized + for WebGPU LLMs but would not fix CPU-only Safari, would add a second model + packaging ecosystem, and is unnecessary for correcting admission control. +- **Transformers.js / ORT remains appropriate** because it already supports + Qwen2 and the existing multi-architecture catalog. The defect was local fit + estimation, not missing framework support. + +## Reproduction and verification + +Automated minimum reproduction in `web/src/models/models.test.ts`: + +1. Select the Qwen2.5 1.5B catalog entry. +2. Evaluate it with no WebGPU adapter and a 2 GiB WASM budget. +3. Assert the verdict is `too-large` (the old implementation returned `tight`). +4. Evaluate with a usable WebGPU adapter and assert q4f16 remains selectable. + +Additional tests ensure the smallest real artifact is used when nothing fits. +The complete local test/quality commands are recorded in PR #18. diff --git a/docs/case-studies/issue-17/issue.md b/docs/case-studies/issue-17/issue.md new file mode 100644 index 0000000..89c7ea9 --- /dev/null +++ b/docs/case-studies/issue-17/issue.md @@ -0,0 +1,16 @@ +# Archived issue #17 + +- URL: https://github.com/link-assistant/model-in-browser/issues/17 +- Title: Qwen 2.5 instruct loading failed +- Author: `konard` +- Created: 2026-06-12 22:32:17 UTC +- Label: `bug` +- Comments at investigation time (2026-07-12): none + +## Original report + +The report consists of the attached screenshot and a request to archive the +evidence, reconstruct the event sequence, identify every requirement and root +cause, research related components, and implement all applicable fixes in one +pull request. The original screenshot is preserved as +`source-assets/issue-17-screenshot.png`. diff --git a/docs/case-studies/issue-17/source-assets/issue-17-screenshot.png b/docs/case-studies/issue-17/source-assets/issue-17-screenshot.png new file mode 100644 index 0000000..1d03227 Binary files /dev/null and b/docs/case-studies/issue-17/source-assets/issue-17-screenshot.png differ diff --git a/web/src/App.tsx b/web/src/App.tsx index d70254c..14bbfc6 100644 --- a/web/src/App.tsx +++ b/web/src/App.tsx @@ -16,6 +16,8 @@ import { } from './models/catalog'; import { detectDeviceCapabilities, + evaluateFitForDtype, + fitsDevice, pickBestDtype, formatBytes, type DeviceCapabilities, @@ -241,12 +243,17 @@ function App() { const caps = capsRef.current; const device: 'webgpu' | 'wasm' = entry.webgpu && caps?.webGpuAdapter ? 'webgpu' : 'wasm'; + const dtype = chosenDtypeFor(entry); + const cpuCaps = caps ? { ...caps, webGpuAdapter: false } : null; loadPayload = { engine: 'transformers', repo: entry.repo, revision: entry.revision, - dtype: chosenDtypeFor(entry), + dtype, device, + allowWasmFallback: + cpuCaps !== null && + fitsDevice(evaluateFitForDtype(entry, cpuCaps, dtype)), ortBase: resolveOrtBase(), }; } else { diff --git a/web/src/models/device.ts b/web/src/models/device.ts index 9ec9ad7..c0953cf 100644 --- a/web/src/models/device.ts +++ b/web/src/models/device.ts @@ -238,7 +238,15 @@ export function evaluateFitForDtype( ): ModelFit { const budget = budgetFor(entry, caps); const usesWebGpu = entry.webgpu && caps.webGpuAdapter; - const runtimeBytes = estimateRuntimeBytes(entry, dtype); + // ORT's WASM backend also keeps allocator/graph memory inside the wasm32 + // heap. Account for that backend-specific headroom on top of the model/KV + // estimate; without it, ~1.5B models are incorrectly advertised as runnable + // near the browser's practical 2 GiB limit and can terminate with Aborted(). + const wasmHeapOverhead = + entry.engine === 'transformers' && !usesWebGpu ? 1.8 : 1; + const runtimeBytes = Math.round( + estimateRuntimeBytes(entry, dtype) * wasmHeapOverhead + ); const dlBytes = downloadBytes(entry, dtype); const budgetFraction = runtimeBytes / budget; @@ -297,7 +305,7 @@ export function pickBestDtype( if (entry.engine !== 'transformers' || entry.variants.length === 0) { return undefined; } - const available = entry.variants.map((v) => v.dtype); + const available = entry.variants.map((variant) => variant.dtype); // Highest quality first. const byQuality = [...DTYPE_ORDER].reverse().filter((d) => available.includes(d)); @@ -312,9 +320,11 @@ export function pickBestDtype( } if (tight) return tight; - // Nothing fits — return the smallest (most compressed) variant. - const smallest = [...DTYPE_ORDER].filter((d) => available.includes(d)); - return smallest[0] ?? available[0]; + // Nothing fits — return the smallest actual artifact. Quantization labels do + // not guarantee file-size order (some q4 graphs are larger than q8). + return [...entry.variants] + .filter((variant) => available.includes(variant.dtype)) + .sort((a, b) => a.bytes - b.bytes)[0]?.dtype; } /** diff --git a/web/src/models/models.test.ts b/web/src/models/models.test.ts index 25b2644..a5f8745 100644 --- a/web/src/models/models.test.ts +++ b/web/src/models/models.test.ts @@ -185,6 +185,27 @@ describe('device fit estimation', () => { expect(small).toBe('q4f16'); }); + it('does not advertise Qwen 2.5 1.5B as runnable in the WASM heap', () => { + const entry = getModelById('qwen2.5-1.5b-instruct')!; + const cpu = makeCaps({ + webGpuAdapter: false, + memoryBudgetBytes: 2 * GB, + }); + + expect(pickBestDtype(entry, cpu)).toBe('q4f16'); + expect(evaluateFit(entry, cpu).level).toBe('too-large'); + }); + + it('keeps q4f16 available when a usable WebGPU adapter exists', () => { + const entry = getModelById('qwen2.5-1.5b-instruct')!; + const gpu = makeCaps({ + webGpuAdapter: true, + gpuBudgetBytes: 2 * GB, + }); + + expect(pickBestDtype(entry, gpu)).toBe('q4f16'); + }); + it('uses the GPU budget when the model and device support WebGPU', () => { const entry = getModelById('smollm2-135m-instruct')!; const caps = makeCaps({ diff --git a/web/src/worker.ts b/web/src/worker.ts index 0868233..4124621 100644 --- a/web/src/worker.ts +++ b/web/src/worker.ts @@ -81,6 +81,8 @@ export interface LoadPayload { dtype?: Dtype; /** Preferred execution device. */ device?: 'webgpu' | 'wasm'; + /** Whether this exact model/dtype fits the WASM heap if WebGPU fails. */ + allowWasmFallback?: boolean; // --- candle engine --- modelUrl?: string; @@ -385,6 +387,11 @@ async function loadTransformersModel(payload: LoadPayload): Promise { } catch (err) { // WebGPU can fail at pipeline-build time on some adapters — fall back to WASM. if (requested === 'webgpu') { + if (payload.allowWasmFallback === false) { + throw new Error( + `WebGPU could not load the ${dtype} model, and it is too large for the WASM/CPU fallback on this device.` + ); + } postMessage({ type: 'status', payload: 'WebGPU unavailable — falling back to CPU (WASM)...',