feat: turboquant optimizations for llama.cpp

[prelegalwonder]

Description

turbo-quant implementation for llama.cpp

Use Case

there is a turbo-quant implementation at https://github.com/unixsysdev/llama-turboquant that might be advantageous to integrate with the other amd-strix-halo optimizations.

The more we can squeeze out of consumer hardware, the better.

Proposed Solution

No response

Alternatives Considered

No response

[Lightheartdevs]

Thanks for flagging this — TQ3_0's KV cache compression is exactly the kind of optimization that matters on unified memory systems like Strix Halo and the new NVIDIA Grace Blackwell (GB10). Getting 4.6x compression on the KV cache with minimal perplexity loss would let users push significantly longer contexts or fit larger models into the same memory budget.

We're absolutely looking into how best to integrate turboquant. It's on the list. The main question we're working through is the right integration path — whether that's as a build option, an alternative backend, or waiting for the techniques to land upstream in llama.cpp. We want to make sure it's maintainable long-term.

Appreciate the suggestion — keep them coming.

[prelegalwonder]

There's also another implementation rotor-quantization that is showing possibly more promise.
https://github.com/scrya-com/rotorquant