FastFlowLM / XDNA2 — Performance Observations & Validation Roadmap

[gatman45]

FastFlowLM / XDNA2 — Performance Observations & Validation Roadmap

(Revised draft — community review requested)

Date: July 23, 2026
Scope: FastFlowLM (FLM) performance observations and XDNA2 architectural analysis.
Status: Draft for review. Numbers below are my own measurements unless a public source is cited; none of this has been validated by the FLM team.

---

TL;DR

XDNA2 provides substantial theoretical compute headroom (~50 TOPS INT8). Current decode throughput measurements suggest that not all of this potential is being translated into observed performance — the cause is not established here, only the symptom is measured.

My instrumentation indicates substantial host-side overhead (CPU orchestration, XRT/IOCTL call counts, context-reconstruction time) that may account for part of the observed gap, decomposed below.

Published papers (TileFuse, STEEL, MLIR-AIR, AMD Agent, "Striking the Balance") show that similar gaps have been closed elsewhere through fusion, pipelining, dispatch reduction, and context/BO reuse. AMD Agent reports 2.2×–4.0× end-to-end speedups on the same NPU family using techniques such as dispatch merging and context reuse (full text verified — see Section 2.2). These results demonstrate that optimizations of this class can materially affect performance, but do not imply that FLM is limited by the same mechanisms.

This document does not cover the 0xc01e0200 TDR crash or the 152-byte BO allocation workaround. That is a separate, narrower finding — see #242 for that specific discussion. Conflating the two undermines both.

Everything below is labeled by evidence strength. Where I claim a number, I say how I got it.

Community review requested: to identify incorrect assumptions, missing evidence, alternative explanations, or additional experiments — not to validate the conclusions as-is.

This is an externally derived performance analysis based on external measurements, profiling, and publicly available sources. Its purpose is to identify potential performance gaps, formulate technically testable hypotheses, and propose reproducible experiments to validate or refute them.

---

Legend

Label	Meaning
[CONFIRMED]	Public documentation or a directly reviewed source explicitly supports the stated fact.
[CONFIRMED PRINCIPLE]	A general engineering/architectural principle supported by multiple independent sources, as opposed to one specific fact or measurement.
[VALID IN CONTEXT]	Supported by a published source within that source's specific experimental setup; not generalized to FLM.
[MY MEASUREMENT]	Observation from my own external instrumentation (see Methodology below). Not FLM-internal telemetry. Reproducible by anyone who runs the same instrumentation.
[PLAUSIBLE]	Technically coherent interpretation consistent with current observations, but lacking direct validation.
[EXTRAPOLATION]	Used as an external point of comparison rather than evidence about FLM.
[NOT PROVEN]	Could not be independently established from the sources reviewed.
[HYPOTHESIS]	Technically plausible explanation requiring controlled validation.

Note: The original draft used "[FLM INTERNAL MEASUREMENT]" for the numbers below. That was a poor choice of words — it reads as if I had access to FastFlowLM's own internal telemetry or source-level profiling, which I don't. Renamed to [MY MEASUREMENT] throughout, with methodology stated up front.

---

Methodology for [MY MEASUREMENT] entries

All measurements below were obtained on the following setup unless otherwise noted:

Hardware:

• SoC: AMD Ryzen AI 9 365 (Strix Point), NPU device ID 0x17F0
• RAM: 32 GB DDR5
• Storage: NVMe SSD

Software:

• OS: Windows 11 Pro (Build 26200)
• FLM version: v0.9.45 (official binary)
• XRT: version provided with FLM
• NPU driver: IpuMcdmDriver v32.0.203.329 (updated since the driver reported in Unable to run gpt-oss 20b on 32GB machine #242, v32.0.203.314 — this measurement session used a newer driver build)

Instrumentation:

• XRT call tracing: Custom wrapper DLL (LD_PRELOAD-style on Windows using Microsoft Detours) that intercepts xrt_coreutil.dll exports and logs entry/exit timestamps with QueryPerformanceCounter. The wrapper logs every call to xrtBOAlloc, xrtBOMap, xrtBOSync, xrtRunExec, xrtRunWait, and associated functions — capturing call count, duration, and argument sizes. This produces a per-token timeline of XRT activity.
• IOCTL capture: ETW (Event Tracing for Windows) tracing of the NPU driver's ETW provider (exact provider name not independently confirmed at time of writing), collected via xperf / wpr during FLM inference. IOCTL command codes and return values are decoded from the ETW stream.
• DMA timing: Derived from the XRT wrapper's xrtBOSync entry/exit pairs, cross-referenced with ETW DMA completion events.
• Power measurement: hwmonitor polling at 100 ms intervals via the APU power management interface.
• TTFT decomposition: Custom timer injected into FLM's HTTP request path via a local proxy recording wall-clock time between POST /v1/chat/completions and the first token output.

Trace data:

• xrt_trace.csv: ~205 000 lines, covering ~15 inference steps with full XRT call log.
• ETW trace files (.etl): ~50 MB per capture session.

Reproducibility note: All measurements were taken on the specific hardware/software config above. Results will vary with different FLM versions, driver versions, memory configurations, and workload parameters. The instrumentation tools (wrapper DLL + ETW) can be rebuilt from the descriptions above by anyone with equivalent hardware — contact me for detailed build instructions.

---

1. Executive Summary

Claim: XDNA2 provides substantial compute capability for AI inference.
Status: [CONFIRMED]
Evidence: "Striking the Balance" (arXiv:2512.13282) reports measured performance of up to 38.05 TOPS INT8 in its optimized GEMM evaluation on XDNA2, and up to 14.71 TOPS in bf16. Independent optimization experiments (Estévez, May 2026, destevez.net) achieved 56 TOPS INT8 peak with custom mlir-aie kernels on the same NPU architecture, confirming the hardware ceiling is in the 50-56 TOPS range under ideal conditions. Architectural references indicate a theoretical peak in the ~50 TOPS INT8 range for the 4×8 compute topology.
Refinement: The 38.05 TOPS figure is a benchmark result under that paper's specific GEMM workload — not a guaranteed end-to-end LLM inference throughput. LLM decode is often GEMV-dominated and bound by memory movement, synchronization, and host/runtime overhead rather than raw arithmetic throughput.

Claim: Current software implementations can operate substantially below XDNA2's theoretical potential for LLM workloads.
Status: [PLAUSIBLE], principle confirmed by literature.
Evidence: MLIR-AIR (arXiv:2510.14871), AMD Agent, TileFuse, STEEL, and related work emphasize spatial mapping, tiling, memory locality, scheduling, synchronization, kernel fusion, and asynchronous execution as first-order performance factors on this class of hardware. The OllamaAMDNPU project (github.com/BrandedTamarasu-glitch/OllamaAMDNPU) provides a practical open-source backend demonstrating similar optimization challenges. The AMD IRON library (github.com/amd/iron) offers pre-built operators that show what's possible with a close-to-metal NPU API.
Refinement: This does not establish that FLM's gap is caused by software alone, or by any single factor. The hypothesis is that the gap is a combination of runtime orchestration, dispatch granularity, context management, spatial utilization, memory movement, operator fusion, synchronization, and workload characteristics.

---

2. Architectural and Performance Observations

2.1 Sequential CPU Orchestration

Claim: CPU-side dispatch and orchestration consume ~60 ms/token (~54% of observed execution time), while effective NPU compute is ~5 ms/token.
Status: [MY MEASUREMENT] — see Methodology section for how this was captured.
Interpretation: If these traces represent the full critical path, this points to a real optimization opportunity in the host/runtime control plane. This alone doesn't establish that all CPU time is avoidable, or that CPU orchestration is the sole cause of the gap.
Recommended validation: Correlate CPU execution, XRT calls, IOCTL activity, synchronization points, DMA transfers, command submission, and tile activity on one timeline.

Claim: Conventional layer-by-layer execution can introduce substantial overhead on spatial AI accelerators.
Status: [VALID IN CONTEXT]
Evidence: STEEL (arXiv:2607.09385, verified — accepted at IEEE COINS 2026) analyzes this in its own experimental environment: an average 22.8× improvement over layer-by-layer execution on XDNA2, and a 9.6× latency reduction over prior state of the art on XDNA1, measured on an AMD Ryzen AI 9 HX 370 SoC.
Refinement: This is evidence that execution granularity and pipelining can matter a lot — not proof that the same magnitude applies to FLM. It should not be quoted as an expected FLM speedup.

2.2 Operator Fusion and Dispatch Granularity

Claim: Separately executed operators increase dispatch count, data movement, synchronization points, and intermediate memory operations.
Status: [CONFIRMED PRINCIPLE], consistent with MLIR-AIR, TileFuse, and AMD Agent (which reports a direct 15→3 per-layer dispatch reduction, full text verified).
Refinement: Exact impact depends on operator boundaries, tensor sizes, memory locality, and the specific runtime implementation — needs direct FLM measurement.

Claim: Inter-CU gaps represent ~27–67% of NPU time per token, with ~21 ms of observed DMA stall.
Status: [MY MEASUREMENT]
Interpretation: Suggests a real pipeline-overlap opportunity, if the profiling methodology correctly separates compute, DMA, and idle time. Doesn't by itself prove operator fusion is the sole cause.

Claim: TileFuse fuses weight unpacking, dequantization, and GEMM/GEMV computation into one execution path, reporting ~+121.6% (GEMM) and up to ~+281% (GEMV) at kernel level, more than 2× performance/energy efficiency over strong iGPU baselines on GEMM, and — in end-to-end LLM experiments on Ryzen AI laptops — up to 2.0× lower prefill latency with over 64% lower energy consumption.
Status: [VALID IN CONTEXT] (arXiv:2606.11357, verified — full text confirmed, not just abstract)
Refinement: The end-to-end 2.0× prefill figure is a stronger reference point for FLM than the isolated kernel percentages, since it's measured on a full LLM pipeline rather than a single kernel. Still not a predicted FLM gain, but the best available anchor for the W4A16 hypothesis (Section 2.5).

Correction of correction (this revision): A previous revision of this document claimed the "15→3 dispatches" and "XRT context/BO reuse" statements about AMD Agent were unsupported, based on checking only the paper's abstract. Having now read the full text (arXiv:2606.07586, "From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUs," Li, Wang, Zhang, Bayliss — Cornell/AMD), both original claims are confirmed accurate. Section III-B states verbatim: "We merged consecutive kernels into single dispatches... reducing per-layer dispatches from 15 to 3" and "We reused the XRT context across calls (Step 2) and recycled per-layer buffer objects via zero-copy mapping (Step 3)." Apologies for the false correction — the lesson is that checking only an abstract is not sufficient to rule a claim unsupported; full-text verification is required before declaring something fabricated.

Claim: A human-guided AI-agent deployment of Llama-3.2-1B on XDNA2 (Ryzen AI 9 HX 370) achieved 2.2× prefill / 4.0× decode speedup over a hand-optimized IRON baseline (TTFT 1.3s, 10.8 tok/s decode at 2048 tokens), via kernel dispatch merging (15→3 per layer), XRT context reuse, zero-copy buffer object recycling, and skipped redundant host-device transfers. The resulting 8-phase agent skill system then autonomously deployed 8 additional decoder-only LLMs end-to-end (0.5–4h each, minimal human guidance; 3 of 8 matched or exceeded the reference's per-hardware efficiency, measured via a defined η_scale metric).
Status: [CONFIRMED] (arXiv:2606.07586, full text verified)
Refinement: This is now the strongest and most directly relevant reference in this entire document for Priority 2 (persistent runtime state) and Priority 3 (dispatch reduction) — it demonstrates, on the same NPU family, that both mechanisms are real, implementable, and contribute to a real 2.2×/4.0× end-to-end speedup. It does not, on its own, tell us how much of FLM's specific gap these two factors would close — that still needs a direct FLM experiment — but the "no external source supports this" framing used earlier in this document (Sections 2.4, 5) is no longer accurate and is corrected below.

Reconciling AMD Agent with OllamaAMDNPU Phase 11 (they are not in conflict): OllamaAMDNPU's Phase 11 found that reducing dispatch count via larger TILE_M values (2048→14336) on its GEMM kernel had zero effect on throughput — they concluded the bottleneck was AIE kernel compute (0.6% utilization), not dispatch/host overhead. This looks like it contradicts AMD Agent's dispatch-count win, but it doesn't: OllamaAMDNPU only batched more invocations of the same, unmodified GEMM kernel — no distinct operators were merged, no inter-operation memory round-trips were eliminated. AMD Agent's "15→3" instead comes from fusing different kernels together (RMSNorm+QKV projections+RoPE into one kernel; output-projection+SwiGLU FFN into another) — this eliminates intermediate materialization and memory traffic between operations, and the drop in dispatch count is a side effect of that fusion, not the source of the gain itself. Practical consequence for Priority 3 below: the target should be phrased as "fuse distinct AIE kernels together," not "reduce dispatch count" — batching the same kernel into fewer, larger dispatches without fusing operators is not expected to help, per OllamaAMDNPU's own negative result.

2.3 Windows / MCDM / XRT / Driver Execution Path

Claim: The current FLM Windows execution path may introduce measurable orchestration overhead.
Status: [MY MEASUREMENT] + [PLAUSIBLE]
Evidence: My profiling observes substantial XRT call activity and IOCTL activity during execution. The Linux AMD XDNA driver documents mechanisms for command submission, mailbox communication, ERT-related execution, and command chaining.
Refinement: The existence of these mechanisms on Linux doesn't by itself prove the Windows MCDM/runtime path is slower. A direct Windows-vs-Linux comparison with equivalent workloads and command traces would be needed.

Claim: ~3064 XRT calls, ~50 IOCTLs, and ~205 ms of cumulative measured latency per token.
Status: [MY MEASUREMENT]
Interpretation: If accurate and on the critical path, this is a meaningful optimization target. The decode throughput deficit observed by other users on Kraken Point hardware (#629: ~50% of official benchmarks, #625: ~3× slower than published) suggests the dispatch overhead pattern may not be specific to this test configuration. Should be decomposed into: CPU-side API overhead, IOCTL submission time, driver processing time, mailbox communication, command scheduling, DMA setup, synchronization/wait, and actual NPU execution.

Claim: The Linux XDNA driver supports command chaining to reduce submission granularity.
Status: [PLAUSIBLE] / requires direct source verification.
Refinement: Specific identifiers should only be presented as confirmed once checked against the exact driver and firmware version in use. The general principle (batching reduces submission overhead) stands independent of the exact identifier names.

2.4 Persistent Context and Initialization Overhead

Claim: ~2362 ms of per-request overhead (~59% of observed TTFT), initially attributed to "context reconstruction."
Status: [MY MEASUREMENT] — the overhead is real, but the label "context reconstruction" was an assumption.
Correction (July 23, 2026): Source code analysis of the FLM orchestrator (v0.9.45, open-source on GitHub) shows that xrt::device and AutoModel are created once and kept alive across HTTP requests — the C++ orchestrator does NOT recreate hardware context per request as previously speculated. The measured 2362 ms therefore originates from inside the closed-source model DLLs (qwen3_5vl_npu.dll, dequant.dll, etc.), not from the orchestrator's device/context management. The exact source remains unidentified.

Interpretation: The 2362 ms is real but its cause is not yet isolated. It could be any of: overlay reloading inside the DLL, weight staging, instruction buffer reconstruction, runtime state initialization, or a cold-start path that runs once at first request. This should be decomposed by instrumenting the DLL boundaries.

Claim: Persistent runtime state (context/BO reuse across requests) is a plausible way to cut initialization overhead.
Status: [VALID IN CONTEXT] — AMD Agent (arXiv:2606.07586, full text) explicitly implements and credits this: "We reused the XRT context across calls" and "recycled per-layer buffer objects via zero-copy mapping," as part of the combination of changes behind its 2.2×/4.0× speedup (see Section 2.2).
Refinement: Real precedent on the same NPU family — but since the FLM orchestrator already keeps the device context alive, this particular optimization path may yield less reduction in FLM than AMD Agent's baseline (which was built on IRON, not FLM). The decomposition experiment is still needed to determine how much of the 2362 ms is addressable vs. structural.

2.5 Q4NX / W4A16 Weight Path

Claim: The FLM Q4NX path stores weights in packed format with dequantization outside the NPU compute path; observed footprint ~7.5 GB vs. ~4.5 GB theoretical 4-bit size.
Status: [MY MEASUREMENT] on this specific build/config — not a general property of Q4NX.
Refinement: The gap should be decomposed into raw packed storage, alignment/padding, metadata, scales, zero points, temporary buffers, dequantized staging copies, and other runtime allocations.

Claim: TileFuse's W4A16 approach integrates unpacking, dequantization, and computation.
Status: [VALID IN CONTEXT]
Refinement: Relevant reference for investigating whether FLM could skip materializing a full higher-precision copy before NPU compute.

Claim: Any W4A16-style execution path is bounded by XDNA2's bf16 compute throughput, not its int8 throughput.
Status: [CONFIRMED] — "Striking the Balance" (arXiv:2512.13282) reports up to 14.71 TOPS bf16 on XDNA2, versus 38.05 TOPS int8.
Refinement: Sets a more realistic ceiling for Priority 4 (W4A16 native execution) than the int8 figures quoted elsewhere in this document. A fused W4A16 path could still beat FLM's current Q4NX path on latency/memory, but shouldn't be expected to approach int8 GEMM throughput.

2.6 DMA / Compute Overlap and Double Buffering

Claim: Incomplete overlap between DMA transfers and compute.
Status: [MY MEASUREMENT] + [HYPOTHESIS]
Evidence: ~21.93 GB/s effective DDR5 bandwidth observed. "Striking the Balance" and MLIR-AIR describe execution models where data movement and compute can overlap.
Refinement: The bandwidth number alone doesn't prove overlap is missing — needs a timeline of DMA engine occupancy vs. compute tile occupancy vs. synchronization dependencies.

Claim: Double-buffered (ping-pong) execution can improve throughput when transfer and compute overlap.
Status: [CONFIRMED PRINCIPLE] — standard pipelining strategy.

---

3. Spatial Utilization and Tile Mapping

Claim: Actual utilization of XDNA2 compute resources in FLM is currently uncharacterized.
Status: [NOT PROVEN / MEASUREMENT GAP]
Interpretation: A 4×8 compute topology being present doesn't mean every FLM operation uses all tiles efficiently. Candidate causes: insufficient parallelism, tile mapping constraints, tensor dimensions, GEMV-dominated decode, memory bandwidth limits, synchronization barriers, DMA dependencies, partial column allocation, workload imbalance, kernel launch granularity.

Recommended experiment: Per-tile/per-column utilization heatmap for prefill, decode, GEMM-heavy layers, GEMV-heavy layers, attention, MLP, and quantized weight loading.

---

4. Structure Size / Allocation / ABI Verification — cross-reference only

(This section intentionally does not re-derive the 152-byte Buffer Object finding — that belongs in #242, where the crash, the tested allocation sizes, and the stability results are documented directly against the TDR error.)

The general open question: some observed sizes in the data path don't match naive expectations. Possible explanations: structure size differences, alignment requirements, ABI differences, wrapper structures, version-specific definitions, padding, allocation granularity, serialization format, driver/runtime interface differences, or an incorrect assumption about the relevant API structure.

Recommended validation:

• Record the exact structure definition and sizeof() / alignof() values
• Record compiler, ABI, XRT, driver, firmware versions
• Trace the allocation size across every API boundary
• Compare expected vs. actual layout byte-by-byte

---

5. Architectural Recommendations

Priority	Area	Description	Key Reference
P1	Column & tile utilization	Measure requested vs. allocated vs. active columns, per-tile occupancy	—
P2	Persistent runtime state	Reuse contexts, BOs, overlays across requests	AMD Agent (confirmed: XRT context reuse + zero-copy BO recycling, part of their 2.2×/4.0× gain)
P3	AIE kernel fusion (not mere dispatch batching)	Merge distinct kernels (e.g. RMSNorm+QKV+RoPE) into one compute kernel to cut inter-op memory round-trips; dispatch-count reduction is a side effect, not the mechanism	AMD Agent (confirmed: 15→3 dispatches/layer via kernel fusion), TileFuse
P4	W4A16 native execution	Compare current Q4NX vs. fused unpack+dequant+compute	TileFuse (arXiv:2606.11357)
P5	DMA/compute pipelining	Sequential vs. ping-pong vs. deeper pipelining	MLIR-AIR
P6	Attention & decode optimization	Separate strategies for prefill vs. decode, GEMM vs. GEMV	STEEL, Zen-Attention

---

6. Gain Table — Expectations, Not Predictions

None of the following are validated FLM performance predictions.

Direction	Basis	Status
Persistent context	~2362 ms measured on per-request overhead (not hw_context recreation — FLM orchestrator already keeps the device alive; the overhead is inside the closed-source DLLs and remains unidentified)	[MY MEASUREMENT] — achievable reduction unknown until the source is isolated
Kernel fusion (not dispatch batching alone)	AMD Agent: 15→3 dispatches/layer via merging distinct kernels (RMSNorm+QKV+RoPE; output-proj+SwiGLU), part of its 2.2×/4.0× gain (confirmed, full text)	[VALID IN CONTEXT] — real mechanism, magnitude not isolated from the other changes bundled with it
Dispatch batching alone, no kernel fusion	OllamaAMDNPU Phase 11: TILE_M 2048→14336 on the same unmodified GEMM kernel — zero throughput change, confirmed compute-bound at 0.6% AIE utilization	[VALID IN CONTEXT] — this is why P3 must target kernel fusion specifically, not just bigger/batched dispatches of unchanged kernels
W4A16 + fused kernels	TileFuse's published GEMM/GEMV gains	[EXTRAPOLATION] — needs FLM A/B test
DMA/compute overlap	~21.93 GB/s effective bandwidth observed	[HYPOTHESIS]
Unified pipeline (STEEL reference)	Up to 22.8× in STEEL's own comparison	[VALID IN CONTEXT]
End-to-end reference: agent-guided deployment	2.2× prefill / 4.0× decode, Llama-3.2-1B on XDNA2 (AMD Agent, arXiv:2606.07586)	[VALID IN CONTEXT] — real, comparable end-to-end number; mechanism not decomposed, not a predicted FLM gain
End-to-end reference: TileFuse	2.0× lower prefill latency, >64% lower energy, TileFuse's own LLM experiments	[VALID IN CONTEXT]

---

7. Critical-Path Analysis — Recommended Next Trace

The single most valuable next measurement is a unified end-to-end critical-path trace:

CPU → Runtime API → XRT → IOCTL → Driver → Mailbox/command submission
    → ERT/scheduling → DMA → Compute tile activity → Synchronization
    → Output transfer → CPU post-processing

The trace should distinguish: work genuinely required, work repeated unnecessarily, work that can be overlapped, work that can be fused, work that can move from host to device, work waiting on memory, and work waiting on synchronization.

---

8. Overall Conclusion

No direct contradiction was found between these observations and the public architectural/research sources reviewed. But the public sources do not independently validate my own measurements, and they don't establish the root cause of FLM's performance characteristics on their own.

Working hypothesis: FLM's performance may be affected by a combination of host/runtime control overhead, dispatch granularity, context initialization, spatial resource utilization, column allocation, per-tile workload distribution, memory movement, DMA scheduling, synchronization, operator fusion opportunities, quantized weight handling, GEMV-specific mapping, and prefill vs. decode workload differences. The relative contribution of each factor is unquantified. This supports a targeted measurement program, not a single-cause explanation.

---

9. Recommended Developer Experiments

1. Per-column and per-tile utilization heatmap
2. Decompose TTFT: context creation, buffer allocation, weight prep/transfer, command construction, NPU execution, output processing
3. Decompose per-token decode latency: CPU orchestration, runtime/API/driver overhead, DMA, NPU compute, synchronization, memory stalls
4. Compare cold-start vs. warm-start execution
5. Prototype persistent runtime state and Buffer Object recycling
6. Test command batching/chaining where supported (expect little gain alone, per OllamaAMDNPU Phase 11 — only useful combined with kernel fusion, see Section 2.2)
7. Prototype fused unpack/dequant/compute kernels
8. Compare current Q4NX execution against a fused W4A16-style path
9. Test DMA ping-pong double buffering
10. Profile prefill and decode independently
11. Separate GEMM-heavy and GEMV-heavy workloads
12. Verify all ABI, structure size, alignment, driver, firmware, runtime version assumptions
13. Correlate CPU, runtime, DMA, and tile activity on a common timeline
14. Evaluate hybrid NPU/iGPU execution only after NPU bottlenecks are quantitatively characterized

---

10. Final Assessment

These observations are a credible basis for further investigation. The public literature supports that spatial mapping, memory locality, dispatch granularity, operator fusion, DMA/compute overlap, persistent runtime state, and low-bit fused execution can all matter — and that workload characteristics strongly influence achievable utilization. It does not justify assigning a definitive percentage of the performance gap to any single factor yet.

Highest-priority next steps: per-column/per-tile utilization measurement, TTFT critical-path decomposition, persistent context/BO reuse, AIE kernel fusion (not mere dispatch batching), DMA/compute overlap, native/fused low-bit weight execution, and separate prefill/decode optimization.

---

References

Source	URL / ID
Striking the Balance (38.05 TOPS int8 / 14.71 TOPS bf16, XDNA2)	arXiv:2512.13282
TileFuse (kernel: +121.6%/+281%; end-to-end: 2.0× prefill latency, >64% less energy) — verified via full HTML text	arXiv:2606.11357
AMD Agent (Li, Wang, Zhang, Bayliss — Cornell/AMD) — full text verified. Confirms: 2.2×/4.0× prefill/decode speedup on Llama-3.2-1B, 15→3 dispatches/layer, XRT context reuse, zero-copy BO recycling, on Ryzen AI 9 HX 370	arXiv:2606.07586
STEEL	arXiv:2607.09385
MLIR-AIR	arXiv:2510.14871
Estévez — 56 TOPS peak	https://destevez.net/2026/05/getting-peak-tops-on-a-ryzen-ai-7-350-npu/
OllamaAMDNPU	https://github.com/BrandedTamarasu-glitch/OllamaAMDNPU
OllamaAMDNPU Changelog	https://brandedtamarasu-glitch.github.io/OllamaAMDNPU/xdna-npu/
AMD IRON / MLIR-AIE	github.com/amd/iron
FLM Issue #242 (TDR crash)	#242
FLM Issue #629 (decode ~50%)	#629
FLM Issue #625 (Qwen3.6 3× slow)	#625
Zen-Attention (attention folding, up to 4× attention-block latency / 32% end-to-end)	arXiv:2508.17593

---

This is an independent, external technical write-up, not an FLM-team-produced document. Every measurement above is mine, taken with my own instrumentation. The goal is to flag credible optimization opportunities and propose concrete, falsifiable experiments — not to claim a bottleneck is already proven. Recommended approach: observe → measure → correlate → hypothesize → test → benchmark → verify.

[astrelsky]

What do you get from xrt-smi validate? It should look something like this.

Validate Device           : [00c8:00:01.1]
    Platform              : NPU Strix
    Power Mode            : Performance
-------------------------------------------------------------------------------
Test 1 [00c8:00:01.1]     : gemm
    Details               : TOPS: 51.3
    Test Status           : [PASSED]
-------------------------------------------------------------------------------
Test 2 [00c8:00:01.1]     : latency
    Details               : Average latency: 65.0 us
    Test Status           : [PASSED]
-------------------------------------------------------------------------------
Test 3 [00c8:00:01.1]     : throughput
    Details               : Average throughput: 62533.0 op/s
    Test Status           : [PASSED]
-------------------------------------------------------------------------------
Validation completed. Please run the command '--verbose' option for more details

[gatman45]

One thing I should flag: the ~60 ms CPU dispatch and ~3064 XRT calls were measured via a Detours wrapper around xrt_coreutil.dll. This wrapper adds its own latency to every intercepted call, and I have no control measurement to subtract that overhead. So the real FLM numbers are lower — possibly significantly.The only measurement in the document that avoids this problem is the effective bandwidth (21.93 GB/s), which is pure arithmetic from TPS × model weight. I’d recommend focusing on that figure and on the TTFT from HTTP timing rather than the hook-derived numbers.
XRT : 2.21.0
Driver : 32.0.20101.3760
Firmware : 1.1.2.64

Test Résultat Valeur
──────────── ───────── ────────────
GEMM INT4 PASSED 51.3 TOPS
Latence — 67 µs avg
Throughput — 60 207 op/s
astrelsky a :

• GEMM : 51.3 TOPS
• Latence : 65 µs
• Throughput : 62 533 op/s

[astrelsky]

XRT : 2.21.0
Driver : 32.0.20101.3760
Firmware : 1.1.2.64

Test Résultat Valeur
──────────── ───────── ────────────
GEMM INT4 PASSED 51.3 TOPS
Latence — 67 µs avg
Throughput — 60 207 op/s
astrelsky a :

• GEMM : 51.3 TOPS
• Latence : 65 µs
• Throughput : 62 533 op/s

Hmm ok, that eliminates that possibility, the device seems to be functioning properly at least.

[gatman45]

Thanks, that's helpful. So the hardware/XRT baseline appears to be healthy on both systems.

That makes the FLM-side measurements more interesting, especially the clean HTTP TPS/TTFT measurements. I'll keep the hook-derived numbers explicitly marked as non-causal until I have a proper control measurement.

Given your experience with XRT/IRON and XDNA2, I'd really appreciate your opinion on where you would look next. Based on the numbers I've shared, do you see any particular bottleneck or measurement I should prioritize to determine where FLM is losing performance?

I'm especially interested in whether you think I should focus first on kernel execution, memory movement, dispatch/runtime overhead, or something else entirely.

[Tagman45]

C:\Program Files\flm>flm bench qwen3.5:9b
[FLM] Fetching models from: C:\Program Files\flm\model_list.json
[FLM] Configuring NPU Power Mode to performance (flm default)
[FLM] Loading model: C:\Users\videl.flm\models\Qwen3.5-9B-NPU2
[FLM] Starting benchmark with 6 stages...
[FLM] Starting benchmark for 32k and iteration 1...
[FLM] Creating checkpoint at context length 30953

[FLM] Model RAW Output:
The following is a summary of the information provided:

• The user wants to know how an AI assistant can interact with them.
• To answer this,
  [FLM] TTFT: 96.9735s, Prefill Speed: 319.475 tokens/s, Decoding Speed: 4.14872 tokens/s
  [FLM] Starting benchmark for 16k and iteration 1...
  [FLM] Creating checkpoint at context length 15481

[FLM] Model RAW Output:
The image shows a woman with dark hair, wearing glasses and a light-colored top.

It appears you've asked to "solve" this puzzle by applying the
[FLM] TTFT: 44.7892s, Prefill Speed: 345.828 tokens/s, Decoding Speed: 5.38089 tokens/s
[FLM] Starting benchmark for 8k and iteration 1...
[FLM] Creating checkpoint at context length 7745

[FLM] Model RAW Output:
It appears you've shared the same story, "The Reclaimer," multiple times in your message. The narrative is complete and compelling on its own! If
[FLM] TTFT: 22.2886s, Prefill Speed: 347.715 tokens/s, Decoding Speed: 4.81852 tokens/s
[FLM] Starting benchmark for 4k and iteration 1...
[FLM] Creating checkpoint at context length 3877

[FLM] Model RAW Output:
The story you provided is a complete and evocative sci-fi narrative titled "The Reclaimer." It follows the journey of Ori, a "reson
[FLM] TTFT: 11.3443s, Prefill Speed: 342.08 tokens/s, Decoding Speed: 4.96108 tokens/s
[FLM] Starting benchmark for 2k and iteration 1...
[FLM] Creating checkpoint at context length 1943

[FLM] Model RAW Output:
This is a compelling sci-fi narrative with strong atmospheric elements and a clear mythos. The story effectively blends the personal journey of Ori (the "Res
[FLM] TTFT: 6.02684s, Prefill Speed: 322.904 tokens/s, Decoding Speed: 5.01686 tokens/s
[FLM] Starting benchmark for 1k and iteration 1...
[FLM] Creating checkpoint at context length 976

[FLM] Model RAW Output:
The floodgates of information had been shut for centuries, and now they were bursting open.

Ori sat in the center of Bastion-9's main
[FLM] TTFT: 3.25071s, Prefill Speed: 300.832 tokens/s, Decoding Speed: 5.04905 tokens/s
[FLM] Starting benchmark for 32k and iteration 2...
[FLM] Creating checkpoint at context length 30953

[FLM] Model RAW Output:
The following is a list of the most common words and phrases used in English:

1. The - "the" is often used to refer to something specific
   [FLM] TTFT: 94.544s, Prefill Speed: 327.547 tokens/s, Decoding Speed: 4.15079 tokens/s
   [FLM] Starting benchmark for 16k and iteration 2...
   [FLM] Creating checkpoint at context length 15481

[FLM] Model RAW Output:
The user's request for an analysis of "the image provided" cannot be fulfilled because no image was uploaded. The prompt includes a series of disconnected, nonsens
[FLM] TTFT: 46.9378s, Prefill Speed: 329.993 tokens/s, Decoding Speed: 5.24395 tokens/s
[FLM] Starting benchmark for 8k and iteration 2...
[FLM] Creating checkpoint at context length 7745

[FLM] Model RAW Output:
It appears you've shared the same story multiple times in your prompt! This is a compelling sci-fi narrative about Ori, a "resonant" who
[FLM] TTFT: 23.6728s, Prefill Speed: 327.444 tokens/s, Decoding Speed: 4.80727 tokens/s
[FLM] Starting benchmark for 4k and iteration 2...
[FLM] Creating checkpoint at context length 3877

[FLM] Model RAW Output:
It appears you've shared the same story, "The Reclaimer," four times in your message. The narrative is a compelling piece of sci-fi fantasy about
[FLM] TTFT: 11.8429s, Prefill Speed: 327.748 tokens/s, Decoding Speed: 4.92551 tokens/s
[FLM] Starting benchmark for 2k and iteration 2...
[FLM] Creating checkpoint at context length 1943

[FLM] Model RAW Output:
The story of Ori, the Reclaimer, is a compelling sci-fi narrative that blends post-apocalyptic survival with high-concept digital mysticism. It effectively
[FLM] TTFT: 6.09455s, Prefill Speed: 319.384 tokens/s, Decoding Speed: 4.98561 tokens/s
[FLM] Starting benchmark for 1k and iteration 2...
[FLM] Creating checkpoint at context length 976

[FLM] Model RAW Output:
The data streams were overwhelming at first—terabytes of history, engineering schematics, poetry from the pre-Fall era, and personal logs of people who had
[FLM] TTFT: 3.36954s, Prefill Speed: 290.386 tokens/s, Decoding Speed: 5.03583 tokens/s
[FLM] Starting benchmark for 32k and iteration 3...
[FLM] Creating checkpoint at context length 30953

[FLM] Model RAW Output:
The following are questions and answers about the history of mathematics, with each question answered in English:

1. What is 0?
   Answer: Zero represents
   [FLM] TTFT: 99.8126s, Prefill Speed: 310.318 tokens/s, Decoding Speed: 4.21557 tokens/s
   [FLM] Starting benchmark for 16k and iteration 3...
   [FLM] Creating checkpoint at context length 15481

[FLM] Model RAW Output:
I apologize, but I cannot fulfill this request. As an AI assistant, my primary function is to assist with information and tasks within the boundaries of safety and ethics
[FLM] TTFT: 47.4057s, Prefill Speed: 326.802 tokens/s, Decoding Speed: 4.58084 tokens/s
[FLM] Starting benchmark for 8k and iteration 3...
[FLM] Creating checkpoint at context length 7745

[FLM] Model RAW Output:
The story you provided is a compelling sci-fi narrative titled "The Reclaimer." It weaves together themes of resurrection, technology, memory, and hope in
[FLM] TTFT: 23.4959s, Prefill Speed: 329.91 tokens/s, Decoding Speed: 4.80333 tokens/s
[FLM] Starting benchmark for 4k and iteration 3...
[FLM] Creating checkpoint at context length 3877

[FLM] Model RAW Output:
It appears you have pasted the same story four times in a row. The narrative of "The Reclaimer" is complete and compelling as written, so there
[FLM] TTFT: 11.8403s, Prefill Speed: 327.895 tokens/s, Decoding Speed: 4.91158 tokens/s
[FLM] Starting benchmark for 2k and iteration 3...
[FLM] Creating checkpoint at context length 1943

[FLM] Model RAW Output:
This is a compelling sci-fi micro-novel with strong atmospheric elements and a clear mythos. You've successfully blended the "post-apocalyptic survival" genre with
[FLM] TTFT: 6.07053s, Prefill Speed: 320.611 tokens/s, Decoding Speed: 5.00631 tokens/s
[FLM] Starting benchmark for 1k and iteration 3...
[FLM] Creating checkpoint at context length 976

[FLM] Model RAW Output:
The floodgates of information had been sealed for too long, and now they burst open all at once.

Ori stood on the observation deck of Bastion
[FLM] TTFT: 3.27423s, Prefill Speed: 298.858 tokens/s, Decoding Speed: 5.04502 tokens/s
[FLM] Starting benchmark for 32k and iteration 4...
[FLM] Creating checkpoint at context length 30953

[FLM] Model RAW Output:
The following is a comprehensive analysis of the provided text, focusing on identifying its structure and content.

Analysis of Input Text: "The Following Is A Comprehensive

[FLM] TTFT: 99.2905s, Prefill Speed: 311.948 tokens/s, Decoding Speed: 4.18124 tokens/s
[FLM] Starting benchmark for 16k and iteration 4...
[FLM] Creating checkpoint at context length 15481

[FLM] Model RAW Output:
The answer is: A.

Wait, I need to re-read the prompt carefully. The user asked for a "long text" that describes something about the
[FLM] TTFT: 47.3873s, Prefill Speed: 326.912 tokens/s, Decoding Speed: 4.57906 tokens/s
[FLM] Starting benchmark for 8k and iteration 4..

PS C:\Users\videl\Desktop> xrt-smi validate --device 0065:00:01.1 --verbose
Verbose: Enabling Verbosity
Validate Device : [0065:00:01.1]
Platform : NPU Strix
Power Mode : Performance

Test 1 [0065:00:01.1] : gemm
Description : Measure the TOPS value of GEMM INT8 operations
Details : Total Duration: 122.7 ns
Average cycle count: 222.0
TOPS: 51.3
Test Status : [PASSED]

Test 2 [0065:00:01.1] : latency
Description : Run end-to-end latency test
Details : Average latency: 85.0 us
Test Status : [PASSED]

Test 3 [0065:00:01.1] : throughput
Description : Run end-to-end throughput test
Details : Average throughput: 53849.0 op/s
Test Status : [PASSED]

Validation completed

ram 5600mt/s