Bug on 0.9.42 on Linux: crash when loading the model

[rmanicardi]

Context

I've build FastFlow 0.9.40 on Linux (Fedora 44) to use Qwen3.5 4B on an AMD Ryzen 7 AI 350 using the build from https://github.com/RaymondKroon/FastFlowLM-Fedora-build (with the only change from Fedora 43 to Fedora 44 in the Dockerfile).

Since there was an issue in triggering the tools I've recompiled v 0.9.42 but flm server was crashing at first request (I'm using Lemonade server as proxy).

I've analyzed the issue with Claude which found the root cause and I've tested a solution. I've tested on my machine and it's working.

I'm not reporting as bug as I don't know if it's better to apply the patch attached or to modify the libllama_npu.so and libqwen2_npu.so libraries.

If you agree the patch is the correct way I can submit the PR.

---

Summary

flm serve crashes during model loading (0.9.42) or on the first inference request, due to heap corruption caused by a missing virtual destructor in the npu_cmd base class. The bug is latent in all prior versions; it became fatal in 0.9.42 because npu_write_cmd gained several data fields, growing from ~8 bytes to 48 bytes.

Affected versions

Version	Behaviour
0.9.42	Fatal — npu_write_cmd is now 48 bytes; the wrong-sized delete corrupts glibc heap metadata, causing SIGABRT at the next unrelated allocation

Steps to reproduce

flm serve qwen3.5:4b --ctx-len 65536

Process terminates with SIGABRT before serving any request (startup crash) or on the first inference call, depending on which call site triggers first.

Root cause

npu_sequence owns a heterogeneous command list:

// src/include/npu_utils/npu_instr_utils.hpp
class npu_sequence {
    std::vector<std::unique_ptr<npu_cmd>> cmds;
    ...
};

npu_cmd declares several pure virtual methods but no virtual destructor:

// src/include/npu_utils/instr_utils/npu_cmd.hpp
struct npu_cmd {
    virtual int print_cmd(uint32_t*, int, int) = 0;
    virtual void to_npu(std::vector<uint32_t>&) = 0;
    virtual void dump_cmd(uint32_t*) = 0;
    virtual int get_op_lines() = 0;
    // virtual ~npu_cmd() = default;  ← MISSING
};

npu_write_cmd is a concrete derived class with 48 bytes of data fields. When cmds.clear() (or the npu_sequence destructor) destroys these elements through unique_ptr<npu_cmd>, C++ invokes the base-class destructor and operator delete(ptr, sizeof(npu_cmd)) = operator delete(ptr, 8) — wrong-sized deallocation of a 48-byte object.

The situation is compounded by the fact that libllama_npu.so and libqwen2_npu.so were compiled at different times against different revisions of the shared headers, so they carry different assumptions about npu_write_cmd's layout — a classic ODR (One Definition Rule) violation.

This is undefined behaviour per C++ [expr.delete] §3: deleting through a base pointer without a virtual destructor is UB when the dynamic type differs.

AddressSanitizer call stacks collected on Fedora 44 / GCC 14:

Startup crash (model loading):

new-delete-type-mismatch: allocated 48 bytes, deallocated as 8 bytes
  delete path:
    npu_sequence::clear_cmds()                    (libqwen2_npu.so)
    qwen3_5vl_npu_sequence::_gen_sequence()       (libqwen3_5vl_npu.so)
    qwen3_5vl_npu::Impl::set_context_length()     (libqwen3_5vl_npu.so)
    Qwen3_5VL::load_model()                       (flm)
  allocation path:
    npu_sequence::rtp_write()                     (libllama_npu.so)

I build FastFlowLM for my Fedora 44 on an AMD Ryzen 7 AI 350 frpm this repository:

https://github.com/RaymondKroon/FastFlowLM-Fedora-build

I've first tried with v 0.9

Inference crash (prefill):

new-delete-type-mismatch: allocated 48 bytes, deallocated as 8 bytes
  delete path:
    Gemm::Impl::generate_seq(...)                 (libqwen2_npu.so)
  allocation path:
    Gemm::Impl::generate_seq(...)                 (libqwen2_npu.so)

Fix

Add a virtual destructor to npu_cmd in src/include/npu_utils/instr_utils/npu_cmd.hpp:

struct npu_cmd {
    virtual ~npu_cmd() = default;   // ← add this line

    virtual int print_cmd(uint32_t*, int, int) = 0;
    virtual void to_npu(std::vector<uint32_t>&) = 0;
    virtual void dump_cmd(uint32_t*) = 0;
    virtual int get_op_lines() = 0;
};

All pre-built NPU libraries must be recompiled against the updated header. The bug exists in any library that was compiled without the virtual destructor; recompiling with it adds the destructor slot to the vtable and restores correct sized-deallocation through unique_ptr<npu_cmd>.

A patch is attached (0001-fix-missing-virtual-destructor-in-npu_cmd.patch).

Workaround (until libraries are recompiled)

For distributors who cannot recompile the pre-built .so files: override npu_sequence::clear_cmds() with a strong symbol that bypasses virtual dispatch, using uptr.release() + ::operator delete() to free each element directly via free(). This is safe because glibc's free() reads the real chunk size from the malloc header, and all known npu_cmd subclasses hold only trivially-destructible data.

0001-fix-missing-virtual-destructor-in-npu_cmd.patch

[ZaneNi]

@rmanicardi Thanks for your feedback!

Sure — feel free to submit a PR! We’ll test it and merge it if everything looks good.

[rmanicardi]

@rmanicardi Thanks for your feedback!

Sure — feel free to submit a PR! We’ll test it and merge it if everything looks good.

Done, I've tested the issue is also on 0.9.3 and created the PR #580