Skip to content

fix(onnx): pass externalData to session_options when using customCache in Node.js#1600

Open
s-zx wants to merge 2 commits intohuggingface:mainfrom
s-zx:fix/1408-customcache-external-data-node
Open

fix(onnx): pass externalData to session_options when using customCache in Node.js#1600
s-zx wants to merge 2 commits intohuggingface:mainfrom
s-zx:fix/1408-customcache-external-data-node

Conversation

@s-zx
Copy link

@s-zx s-zx commented Mar 21, 2026

Problem

When a model uses external data (e.g. model.onnx_data) and the user provides a customCache (which returns a Uint8Array/Response rather than a filesystem path), loading fails in Node.js with:

file_size: system cannot find the specified path: "model.onnx_data"

Root Cause

session.js only sets session_options.externalData when !IS_NODE_ENV.

In Node.js with useFSCache this is intentional — the ONNX runtime locates .onnx_data files automatically relative to the on-disk model path.

But when customCache is used the data arrives as an in-memory Uint8Array (not a file path), so the runtime has no way to discover the sidecar files on disk.

Fix

Check whether any resolved external-data entry is a {path, data} object (i.e. in-memory bytes from a custom cache). If so, pass externalData to session_options even in Node.js:

// Before
if (externalData.length > 0 && !apis.IS_NODE_ENV) {
    session_options.externalData = externalData;
}

// After
if (externalData.length > 0) {
    if (!apis.IS_NODE_ENV || externalData.some(item => typeof item !== 'string')) {
        session_options.externalData = externalData;
    }
}

Fixes #1408

…he in Node.js

When a model uses an external data file (e.g. `model.onnx_data`) and the
user supplies a `customCache` (which returns a `Uint8Array` / `Response`
rather than a filesystem path), the ONNX runtime in Node.js fails to
locate the external data file with an error like:

    file_size: system cannot find the specified path: "model.onnx_data"

Root cause: `session.js` only sets `session_options.externalData` when
`\!IS_NODE_ENV`.  In Node.js with `useFSCache` this is correct — the ONNX
runtime finds the `.onnx_data` files automatically relative to the model
path on disk.  But when `customCache` is used the data arrives as an
in-memory `Uint8Array` (not a file path), so the runtime has no way to
discover the sidecar files.

Fix: also set `session_options.externalData` in Node.js whenever any of
the resolved external-data entries is not a plain string (i.e. it is a
`{path, data}` object carrying in-memory bytes from a custom cache).

Fixes huggingface#1408
@xenova
Copy link
Collaborator

xenova commented Mar 24, 2026

Hi there 👋 thanks for the PR. Any reason for doing typeof item !== 'string' instead of item instanceof Uint8Array?

More specific type check as suggested by maintainer. Since custom cache
returns Uint8Array (binary data) vs string (file paths), checking for
the exact type is clearer and more correct.
@s-zx
Copy link
Author

s-zx commented Mar 25, 2026

@xenova Good call — instanceof Uint8Array is more precise. Updated in the latest commit. The intent is specifically to detect in-memory binary data from a custom cache vs. file paths (strings) from the filesystem cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qwen3 embedding with customCache throw error

3 participants