Add support for out-of-tree builtin modules

[antocuni]

No description provided.

[seibert]

I think this looks reasonable. I'm trying to decide if there are any drawbacks to putting the libraries and flags inside the Python registry for the module, rather than setting them externally (like on the spy command line). I'm wondering if setting on the spy command line might be needed if the C library is distributed separate from the SPy module, and its location varies depending on system setup. (Ex: Maybe I'm using the system installation of libxml, maybe I'm using the one in my conda environment?)

[antocuni]

I'm wondering if setting on the spy command line might be needed if the C library is distributed separate from the SPy module, and its location varies depending on system setup. (Ex: Maybe I'm using the system installation of libxml, maybe I'm using the one in my conda environment?)

that's a good point and I don't know what is the best tradeoff.
The following is a kind of "thinking out loud" section:

My reasoning is that "which concrete C lib to link" is a property of the extension module, not of the final consumer.

The CPython parallel is this:

1. I want to build/install numpy from source
2. the numpy configure step (or equivalent) decides where to find blas&co.
3. I build a wheel
4. from now, the C lib choice is fixed and every python interp can import numpy without thinking about it

The parallel doesn't fully apply to spyvm_modules because it's not clear whether we should treat them as source distributions or "compiled" artifacts.
Also, remember that the interpreter needs a .wasm, which is unlikely to be distributed with your system package manager.

In the qrcodegen example, you must go there and type make in order to build the actual lib (which builds the .wasm for wasi and emscripten, the .so, the .a, etc) this is the equivalent of point (3) above, and point (2) is hardcoded in the __init__.

So probably, a fully (over?) engineered system would be:

1. the spyvm_module exist in source form in a github repo or on disk
2. you have an install/build step which creates/links binary artifacts (if needed)
3. the result of (2) is a manifest which contains the path to (1) plus all the cflags, ldflags, .wasm locations, etc. (or if we want the equivalent of a wheel, a zip file containing all of this)
4. the end user refers to (3) when using spy --extra-vm-module

However, I'm not sure that at this stage we need this level of complexity, especially keeping in mind that out-of-tree vm modules might not be the proper long term solution).

I think we can get 90% of the benefits and 1% of complexity by keeping the build info in the __init__, and if you really need to configure them externally you use env variables.

Happy to be convinced otherwise :)

[github-actions]

Documentation Preview
Preview documentation for PR #559.
Last Updated: 2026-06-13 15:00 UTC

[antocuni]

@seibert it took much longer than expected but now I have something which seem to work.
It it still marked as WIP as I haven't fully decided all the small design issues, but the basic concept seems to work.

To run examples/out-of-tree:

1. build the module first:

❯ make -C examples/out-of-tree/spyvm_qrcodegen/`

2. run the demo in the interpreter. This is VERY slow but the slowness comes from the printing loop (it's nested loop which needs to print things pixel-by-pixel, and in full interpreted mode is awfully slow):

❯ spy examples/out-of-tree/demo/main.spy hello

3. compile the demo:

❯ spy build --release examples/out-of-tree/demo/main.spy 
[release] examples/out-of-tree/demo/build/main 

❯ ./examples/out-of-tree/demo/build/main hello
[...]