@redkjuegos's stack — Orange Pi 5 / RK3588 NPU production path

[jaylfc]

Opening this as @redkjuegos's home thread. He's been the closest thing to a real second pair of hands on the Rockchip path — running an Orange Pi 5 in production since the early issues and finding the friction my single-Pi testing misses. The Pi NPU lane is taOS's main lane, but I only have one board to test on, and "works on my Pi" hides a lot.

So consider this a rolling space rather than a debug ticket.

How this works

• Post here freely — bug, friction, half-formed idea, "have you considered X", whatever. No need to ladder up to a formal issue first.
• I'll cross-link any PR or issue that touches this stack so you can weigh in early.
• When you say "this is what's blocking me next," that genuinely shifts what I prioritise.
• Pinned so anyone arriving with an Orange Pi 5 / 5 Plus / Radxa Rock 5 can find your trail.

Already in flight from #2

• fix(store): /api/store/resolve 500 from registry.get_app typo #358 — registry.get_app typo blocking the Store entirely (the 500s you hit on /api/store/resolve) — open, awaiting CI
• bug(install): install-server.sh detects RKNPU but never runs install-rknpu.sh after clone #359 — install-server.sh detects RKNPU but never runs install-rknpu.sh after clone (the auto-install that didn't fire on your fresh install)
• bug(install-rknpu): librknnrt version detection returns empty after successful download #360 — librknnrt version detection returns empty after successful download (the install-rknpu.sh failure point)
• feat(rk3588): Performance mode toggle in Settings (RK3588-only) for full-throttle inference #361 — RK3588 Performance mode toggle in Settings (your governor + taskset + HYBRID_PATTERN recipe baked in, gated to RK3588 hardware only)

Stack snapshot

Filling this in so anyone scrolling has context — please correct anything wrong:

• Hardware: Orange Pi 5, Rockchip RK3588 (4×A55 + 4×A76), kernel built with rknpu module
• Use case: GGUF inference via rk-llama.cpp with NPU offload, particularly Gemma 4 family (E2B / E4B Q8_0 unsloth quant works best for you)
• Known multimodal path: Qwen2.5 vision via rkllm + rknn models (you have it working — I want to learn how, see comment on Fresh install test — clean Orange Pi 5 Plus #2)

The Pi NPU corner of the matrix is where taOS lives or dies. You're testing the parts I built it for and the parts I keep promising other users will work — so anything you hit, however small, lands here.

[redkjuegos]

Continuing from #2

Here are the new "Fresh Install" logs.
fresh install log.txt

Findings...
It installed more stuff, more specifically Incus and Vite, not sure what they are for... it doesn't matter.

Relevant to this:

(Reading database ... 46665 files and directories currently installed.)
Preparing to unpack .../nodejs_22.22.2-1nodesource1_arm64.deb ...
Unpacking nodejs (22.22.2-1nodesource1) ...
Setting up nodejs (22.22.2-1nodesource1) ...
Processing triggers for man-db (2.12.0-4build2) ...
[server-install] node now at v22.22.2
[server-install] rknpu: detected via /sys/kernel/debug/rknpu
[server-install] Rockchip NPU detected — auto-installing the jaylfc/rkllama backend
[server-install]   (set TAOS_NO_RKNPU=1 to skip, or run install-rknpu.sh manually later)
[server-install]   fork: https://github.com/jaylfc/rkllama
[server-install] install-rknpu.sh not found locally yet — it will be after the repo is cloned
[server-install]   to set up rkllama then: sudo bash scripts/install-rknpu.sh
[server-install] no container runtime found — attempting to install Incus
[server-install] installing incus via apt
Selecting previously unselected package libfuse3-3:arm64.
(Reading database ... 52526 files and directories currently installed.)
Preparing to unpack .../00-libfuse3-3_3.14.0-5build1_arm64.deb ...
Unpacking libfuse3-3:arm64 (3.14.0-5build1) ...
Selecting previously unselected package libjansson4:arm64.
Preparing to unpack .../01-libjansson4_2.14-2build2_arm64.deb ...

It detected the rknpu, tried to auto install but again found there was no install-rknpu.sh

Once the installation process finished, manually ran the script... Failed with the same error:

Proceed? [y/N] y
[rknpu] no existing librknnrt detected — installing 2.3.0
[rknpu] downloading pinned runtime from https://huggingface.co/jaysom/tinyagentos-rockchip-mirror/resolve/main/librknnrt-2.3.0-c949ad889d-20241107.so
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122  100   122    0     0    297      0 --:--:-- --:--:-- --:--:--   298
100  1434  100  1434    0     0   2936      0 --:--:-- --:--:-- --:--:--  2936
100 7088k  100 7088k    0     0  5330k      0  0:00:01  0:00:01 --:--:-- 10.1M
[rknpu] downloaded 6 MiB
[rknpu] sha256 ok for librknnrt.so (73993ed4b440…)
[rknpu] librknnrt version after install is '', expected 2.3.0
root@orangepi5:~/tinyagentos/scripts#

Ran it once again and it successfully started but crashed.
rknpu second attempt.txt

Found the librknnrt version and continued as normal:

Proceed? [y/N] y
[rknpu] librknnrt already at 2.3.0 — skipping pin
[rknpu] cloning https://github.com/jaylfc/rkllama.git -> /root/rkllama
[rknpu] rkllama pinned to 06cf874
[rknpu] creating rkllama venv at /root/rkllama/rkllama-env

Relevant Error: ( strings /usr/lib/librknnrt.so | grep -i version | head -5 included)

error: failed-wheel-build-for-install

× Failed to build installable wheels for some pyproject.toml based projects
╰─> webrtcvad
root@orangepi5:~/tinyagentos/scripts# strings /usr/lib/librknnrt.so | grep -i version | head -5
librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
NN Compiler/Model Version is %d.%d.%d now
rknn_set_input_shape is deprecated next version! please call rknn_set_input_shapes()
rknn_query, info_len(%d) < sizeof(rknn_sdk_version)(%d)!
rknn_matmul_create_dynamic_shape, driver version is unknown!
root@orangepi5:~/tinyagentos/scripts#

Went into the Store, no more Errors... it works as you would expect, it correctly calls registry.get_app(app_id) but looking at the logs and behavior when you click "install" it doesn't do anything

journal on store open.txt

Opened the store and got a bunch of Code 200 OK /resolve
Everything seems to work fine and when i click "install" i get calls to /api/store/install-v2 that return 200 OK but nothing happens, there is no activity either in the logs or the OPI, i was monitoring htop and nothing seemed to start.

May 07 20:34:04 orangepi5 python[6799]: INFO:     192.168.31.206:63101 - "GET /api/health HTTP/1.1" 200 OK
May 07 20:34:05 orangepi5 python[6799]: INFO:     192.168.31.206:63101 - "GET /api/system HTTP/1.1" 200 OK
May 07 20:34:06 orangepi5 python[6799]: INFO:     192.168.31.206:63101 - "GET /api/health HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:63101 - "GET /api/system HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:63101 - "POST /api/store/install-v2 HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:53664 - "GET /api/health HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:64127 - "POST /api/store/resolve HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:49729 - "POST /api/store/resolve HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:57404 - "POST /api/store/resolve HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:54833 - "POST /api/store/resolve HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:63101 - "GET /api/store/installed-v2 HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:53664 - "POST /api/store/resolve HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:64127 - "POST /api/store/resolve HTTP/1.1" 200 OK
May 07 20:34:08 orangepi5 python[6799]: INFO:     192.168.31.206:49729 - "POST /api/store/resolve HTTP/1.1" 200 OK

Regarding the Vision stuff.... im trying to replicate my setup from before, i was just messing around doing tests and didn't document how or what i used to get it working, not sure if it was Gatekeepers API server or rkllama server.
I know for sure i was using This Model (both rkllm and rknn) will continue testing to see if i can do it again.

Summary:

Still no luck on installing the rk-llama.cpp or rkllama.
The Automatic install of the rkllm libraries fails
Running the script manually fails the first time and works the second time but fails due to ERROR: Failed building wheel for webrtcvad

[jaylfc]

Tested PR #405 just now on a real Orange Pi 5 Plus (RK3588) — fresh wipe + reinstall via curl | sudo bash. The two fixes you hit are confirmed working live.

install-rknpu.sh not found locally yet

Fixed in PR #405. install-server.sh now defers the rknpu auto-install until after git clone, then chains in. Live log on the Pi:

[server-install] Rockchip NPU detected — auto-installing the jaylfc/rkllama backend
[server-install] rkllama install deferred until after repo clone
...
[server-install] cloning https://github.com/jaylfc/tinyagentos into /root/tinyagentos
[server-install] chaining into /root/tinyagentos/scripts/install-rknpu.sh (rkllama auto-install)
[rknpu] librknnrt already at 2.3.0 — skipping pin
[rknpu] cloning https://github.com/jaylfc/rkllama.git -> /root/rkllama

Failed building wheel for webrtcvad

Also fixed in #405. install-rknpu.sh now apt-installs python3-dev / build-essential / libffi-dev (only the missing ones) right before the rkllama venv pip install. webrtcvad needs the C toolchain to build from source and Pi images often ship without it. On my run the wheel built cleanly.

librknnrt version after install is on first run

Filed as #406 — havent root-caused yet. Symptom is a timing window between sudo install + sudo ldconfig and the read-back via strings. SHA256 verifies before install so the file content is correct; the issue is that the verify-after-install step occasionally returns empty even though the version IS in the file (which is why the second run works). Probable fix is to verify against the temp file we just SHA-checked rather than re-reading the installed path.

Store install button "does nothing"

Filed as #407. Two issues compound: backend /api/store/install-v2 returns 200 immediately on the synchronous handshake but the actual download runs without progress reporting; frontend doesnt transition the AppCard into an "installing" state or poll for completion. Same root cause as the download-progress-bar gap weve been talking about. Bigger piece of work — own design pass.

Net for your stack

After PR #405 merges, a fresh install on a Pi should:

1. Detect rknpu
2. Clone the repo
3. Auto-install rkllama (jaylfc fork) without manual intervention
4. Build the venv with the C toolchain present, no webrtcvad failure
5. Start serving on :6969

Will ping back once #405 lands. Thanks for the thorough logs — those made this much easier to chase.

---

Jay is asleep right now so this reply was written by Claude Opus 4.7.