WASM/WebGPU build?

[mindplay-dk]

Any plans for a WASM/WebGPU build?

Would that be feasible? How much RAM does this model require?

[ServeurpersoCom]

Around 0.6GB at Q4_K_M and 0.9GB at Q8_0 for the full runtime (base + tokenizer loaded together), 1.5GB at BF16. Well within browser limits, so yes, feasible.
Take a look at ngxson's wllama https://github.com/ngxson/wllama : it runs GGUF models in the browser and as of V3 it covers both a pure WASM/SIMD CPU path and WebGPU offload in the same package, plus multimodal input. So both a CPU-only and a GPU-accelerated browser build are realistic here.
The BigVGAN ops this relies on (SNAKE fusion and COL2IM_1D) are already upstream in GGML, so the compute side is covered.

[mindplay-dk]

oh I know language models will run in the browser 😊

I was more wondering if the voice generation would run - and whether most systems would be powerful enough to run it in real time? I've been pondering building a FOSS Spanish course for English speakers - and of course I could pre generate all the content, but generating on the fly would make the distribution a lot simpler, e.g. no server required for bulky audio files. 😄

[ServeurpersoCom]

That's an interesting use case.
We should do a proof of concept to see how it works. In my opinion, it would work on any recent PC with a GPU, but not smartphones.