Skip to content

Unsloth Q4_K_XL model is not supported because support for Q8_0 is missing #1

@vlisivka

Description

@vlisivka

Example:

$ BEBELM_WEIGHTS_FILE=LFM2.5-8B-A1B-UD-Q4_K_XL.gguf bebelm chat
Chat ready. Type a message. Input Ctrl-D or /exit to quit, /clear to reset history.

User> Hello, how are you?


thread 'main' (188491) panicked at /home/vlisivka/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/bebelm-0.3.4/src/kernels/dequant.rs:173:18:
dequantize: unsupported type Q8_0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions