IOCTL regression using ASR and embedding on Krackan Point chips

[danimalquackers]

Hardware: AMD Ryzen AI 350 (Krackan Point) chip with XDNA2 support on a Framework 13 mainboard. Running NixOS with the nix-amd-ai flake to install FastFlowLM, Lemonade, and other backends.

You can see noamsto/nix-amd-ai#79 for earlier discussion and bisecting of this issue. TL;DR: after FastFlowLM 0.9.44, FLM stopped being able to load and run any models on my Krackan Point system, with an IOCTL-related error. The downstream repository author helped me bisect the issue and ruled out it being a difference in kernel version, since downgrading to 0.9.44 on my up-to-date (kernel 7.1.3) system allowed models to run again. Note that the same error appeared regardless of the model chosen, and regardless of whether it was run with sudo or not:

❯ flm run gemma4-it:e4b
[FLM]  Loading model: /home/example/.config/flm/models/Gemma4-E4B-IT-NPU2
[ERROR]  Failed to load model: DRM_IOCTL_AMDXDNA_CREATE_HWCTX IOCTL failed (err=-22): Invalid argument

The kernel logs also included several related lines:

[84319.443131] amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_send_mgmt_msg_wait: command opcode 0x2 failed, status 0x2000003
[84319.443143] amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_hwctx_init: Alloc hw resource failed, ret -22
[84319.443198] amdxdna 0000:c2:00.1: [drm] *ERROR* amdxdna_drm_create_hwctx_ioctl: Init hwctx failed, ret -22

Other relevant output:

❯ flm validate
[Linux]  Kernel: 7.1.3
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.8
[Linux]  Memlock Limit: infinity

c2:00.1 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo Neural Processing Unit [1022:17f0] (rev 20)
	Subsystem: Framework Computer Inc. Device [f111:000b]
	Kernel driver in use: amdxdna
	Kernel modules: amdxdna

Plus some troubleshooting steps from an older kernel and FLM version:

❯ journalctl -k -b | grep -iE 'amdxdna|pasid|carveout|sva'
Aug 05 14:01:09 pike kernel: amdxdna 0000:c2:00.1: [drm] Load firmware amdnpu/17f0_10/npu_7.sbin
Aug 05 14:01:09 pike kernel: amdxdna 0000:c2:00.1: enabling device (0000 -> 0002)
Aug 05 14:01:09 pike kernel: [drm] Initialized amdxdna_accel_driver 0.6.0 for 0000:c2:00.1 on minor 0

❯ cat /sys/bus/pci/devices/0000:c2:00.1/iommu_group/type
─────┬───────────────────────────────────────────────────────────────────────────────
     │ File: /sys/bus/pci/devices/0000:c2:00.1/iommu_group/type
─────┼───────────────────────────────────────────────────────────────────────────────
   1 │ identity
─────┴──────────────────────────────────────────────────────────────────────────────

❯ sudo grep -H . /sys/module/amdxdna/parameters/*
/sys/module/amdxdna/parameters/aie2_max_col:128
/sys/module/amdxdna/parameters/force_cmdlist:Y

[Atomic-Germ]

What are your iommu settings? I have similar hardware and it's working with this version.
Kernel: 7.2.0-0.rc6.260806g0d8395707651.51.fc45.x86_64
NPU FW Version: 1.1.2.64
amdxdna version: 0.10
Memlock Limit: infinity

I don't know if any recent changes would have brokens something with amdxnda 0.80 but I'd look at that.

journalctl -k -b | grep -iE 'amdxdna|pasid|carveout|sva'
Aug 07 08:09:36 fedora kernel: amdxdna 0000:c2:00.1: [drm] Load firmware amdnpu/17f0_10/npu_7.sbin
Aug 07 08:09:36 fedora kernel: amdxdna 0000:c2:00.1: enabling device (0000 -> 0002)
Aug 07 08:09:36 fedora kernel: [drm] Initialized amdxdna_accel_driver 0.10.0 for 0000:c2:00.1 on minor 0

and I have this:

/sys/module/amdxdna/parameters/force_iova:N
/sys/module/amdxdna/parameters/tdr_timeout_ms:2000

EDIT: Are you trying to run two flm run sessions? That will reliably give that exact error.

[simongoudie]

I'm on: Lenovo Yoga 7 Pro, Ryzen AI 350 / Krackan Point, 1022:17f0 rev 20). Kernel 7.1.6, amdxdna 0.8, NPU FW 1.1.2.64., FLM 0.9.46. NixOS/Hyprland

Just adding that I can reproduce the error and log with two running sessions as well (flm run x2, or lemonade+flm run), but I'm running fine when using just the one.

[danimalquackers]

Thanks, that tipped me off to look for other processes. Strangely, this seems to pop up only when using concurrent server mode. I updated everything back to the latest versions and now flm run works.

It looks like I might've had flm serve running in a service, but it wasn't responding so I assumed it had failed. However, what I discovered is that flm serve was running but doesn't seem to be able to load the primary model if both --asr 1 and --embed 1 are set. If either --asr or --embed are set but not both, then the model loads fine, but otherwise it starts up but without the main model:

❯ flm serve "qwen3.5:2b" --embed 1 --asr 1
[FLM]  ASR mode enabled: reserving additional 1GB of memory
[FLM]  Embedding mode enabled: reserving additional 300MB of memory
[FLM]  Using environment-specified port: 52625
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Whisper-V3-Turbo-NPU2
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Embedding-Gemma-300M-NPU2
[FLM]  Qwen3.5 pre-resize image height to 720 pixels if larger than that
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Qwen3.5-2B-NPU2
[ERROR]  Failed to load model: DRM_IOCTL_AMDXDNA_CREATE_HWCTX IOCTL failed (err=-22): Invalid argument
[Error]  Failed to load default model: qwen3.5:2b
[FLM]  Starting server on port 52625...
[FLM]  WebServer started on port 52625 with 10 I/O threads
[FLM]  Press Ctrl+C to stop.

❯ flm serve "qwen3.5:2b" --embed 1
[FLM]  Embedding mode enabled: reserving additional 300MB of memory
[FLM]  Using environment-specified port: 52625
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Embedding-Gemma-300M-NPU2
[FLM]  Qwen3.5 pre-resize image height to 720 pixels if larger than that
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Qwen3.5-2B-NPU2
[FLM]  Starting server on port 52625...
[FLM]  WebServer started on port 52625 with 10 I/O threads
[FLM]  Press Ctrl+C to stop.

❯ flm serve "qwen3.5:2b" --asr 1
[FLM]  ASR mode enabled: reserving additional 1GB of memory
[FLM]  Using environment-specified port: 52625
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Whisper-V3-Turbo-NPU2
[FLM]  Qwen3.5 pre-resize image height to 720 pixels if larger than that
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Qwen3.5-2B-NPU2
[FLM]  Starting server on port 52625...
[FLM]  WebServer started on port 52625 with 10 I/O threads
[FLM]  Press Ctrl+C to stop.

The first fails with the error I was seeing:

ug 08 18:04:34 pike kernel: amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_send_mgmt_msg_wait: command opcode 0x2 failed, status 0x2000003
Aug 08 18:04:34 pike kernel: amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_hwctx_init: Alloc hw resource failed, ret -22
Aug 08 18:04:34 pike kernel: amdxdna 0000:c2:00.1: [drm] *ERROR* amdxdna_drm_create_hwctx_ioctl: Init hwctx failed, ret -22

The other two succeed. For now I'll run with just --asr 1 and no embedding model, but this still seems to be a regression since it used to be possible to use both sidecars.

Other info:

Aug 05 16:50:22 pike kernel: amdxdna 0000:c2:00.1: [drm] Load firmware amdnpu/17f0_10/npu_7.sbin
Aug 05 16:50:22 pike kernel: amdxdna 0000:c2:00.1: enabling device (0000 -> 0002)
Aug 05 16:50:22 pike kernel: [drm] Initialized amdxdna_accel_driver 0.8.0 for 0000:c2:00.1 on minor 0

/sys/module/amdxdna/parameters/aie2_max_col:128
/sys/module/amdxdna/parameters/force_cmdlist:Y
/sys/module/amdxdna/parameters/force_iova:N

[Atomic-Germ]

@danimalquackers I just reproduced the same with flm serve qwen3.6-moe:35b-a3b --embed 1 --asr 1 -- works with one or the other but not both. I'm guessing this and the original two-processes issue are one and the same.
I bet asr and embed try to use the same process; or rather try not to.