Skip to content

fix: prevent Qwen 2.5 WASM out-of-memory loads - #18

Merged
konard merged 4 commits into
mainfrom
issue-17-22f8c7a0f0d2
Jul 26, 2026
Merged

fix: prevent Qwen 2.5 WASM out-of-memory loads#18
konard merged 4 commits into
mainfrom
issue-17-22f8c7a0f0d2

Conversation

@konard

@konard konard commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #17.

Qwen2.5 1.5B was advertised as fitting CPU-only Safari because the model selector estimated ONNX runtime memory from download bytes but omitted ONNX Runtime's additional WASM heap/session-construction overhead. Loading the 1.22 GB q4f16 graph then terminated inside Emscripten with the opaque Aborted() shown in the issue.

This PR:

  • applies backend-aware peak-memory estimation: the existing model/activation estimate is retained for WebGPU, while WASM includes conservative graph/allocator/session heap headroom;
  • classifies Qwen2.5 1.5B as too large on a 2 GiB CPU/WASM budget, while keeping q4f16 available on capable WebGPU devices;
  • permits WebGPU → WASM retry only when the exact model/dtype also passes the CPU fit calculation;
  • chooses the smallest fallback artifact by measured bytes (Qwen's q4 file is larger than q8, so dtype labels are not a safe size ordering);
  • archives the original issue screenshot and provides a complete requirement matrix, timeline, root-cause analysis, upstream research, alternatives, and verification in docs/case-studies/issue-17/;
  • adds the required changelog fragment.

Reproduction

Before this fix:

  1. Open the deployed app in Safari (WASM/CPU backend).
  2. Select Qwen2.5 1.5B Instruct at q4f16.
  3. Observe that the card says the model fits and begins the ~1.1 GiB download.
  4. Session construction ends with Error: Aborted(). Build with -sASSERTIONS for more info.

Original evidence:

Qwen2.5 WASM load abort

Automated regression

web/src/models/models.test.ts now evaluates the real Qwen2.5 1.5B catalog entry with a CPU-only 2 GiB budget and asserts it is too-large; the old code returned tight and allowed the crashing load. A companion assertion verifies q4f16 remains selected with a usable WebGPU adapter.

Verification

Passing locally:

  • npm test --prefix web -- --run — 39 tests
  • ./scripts/build-wasm.sh
  • npm run build --prefix web — TypeScript + production Vite build
  • cargo fmt --check
  • cargo test — 19 unit/integration tests + 3 doctests
  • cargo clippy --all-targets --all-features
  • node scripts/check-file-size.mjs
  • git diff --check

Known repository-wide tooling issue: npm run lint --prefix web cannot start because ESLint 9 is installed but the repository has no eslint.config.js/mjs/cjs. This predates and is unrelated to the changed files; TypeScript compilation and all configured CI checks remain covered above.

Case study

See docs/case-studies/issue-17/ for preserved evidence and the full analysis.

Adding .gitkeep for PR creation (default mode).
This file will be removed when the task is complete.

Issue: #17
@konard konard self-assigned this Jul 12, 2026
@konard konard changed the title [WIP] Qwen 2.5 instruct loading failed fix: prevent Qwen 2.5 WASM out-of-memory loads Jul 12, 2026
@konard
konard marked this pull request as ready for review July 12, 2026 05:26
@konard

konard commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

Working session summary

Implemented and finalized PR #18.

Key changes:

  • Corrected WASM peak-memory estimation so Qwen2.5 1.5B is not offered on unsafe CPU-only devices.
  • Guarded WebGPU-to-WASM fallback using the same memory-fit calculation.
  • Selects fallback variants by actual artifact size.
  • Added regression tests for CPU and WebGPU behavior.
  • Archived the original screenshot and added the full issue case study.
  • Added a changelog fragment and removed the WIP placeholder.

Verification:

  • 39 web tests passed.
  • Production WASM and Vite builds passed.
  • Rust tests, doctests, formatting, Clippy, and file-size checks passed.
  • All fresh GitHub CI, deployment build, and browser E2E workflows passed on commit 5f2a2c9.
  • Worktree is clean; PR is mergeable and ready for review.

This summary was automatically extracted from the AI working session output.

@konard

konard commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Model: GPT-5.6 Sol
  • Provider: OpenAI
  • Public pricing estimate: $14.741969

📊 Context and tokens usage:

  • 281.7K / 200K (141%) input tokens, 22.6K / 128K (18%) output tokens

Total: (281.7K + 10.9M cached) input tokens, 22.6K output tokens, $14.741969 cost

🤖 Models used:

  • Tool: OpenAI Codex
  • Requested: gpt-5.6-sol
  • Thinking level: medium (~15999 tokens)
  • Model: GPT-5.6 Sol (gpt-5.6-sol)

📎 Log file uploaded as Gist (3032KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard

konard commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

✅ Ready to merge

This pull request is now ready to be merged:

  • All CI checks have passed
  • No merge conflicts
  • No pending changes

Monitored by hive-mind with --auto-restart-until-mergeable flag

@konard
konard merged commit 9044b1f into main Jul 26, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen 2.5 instruct loading failed

1 participant