Inference acceleration

[CQUPT-CaiKe]

Excellent work. I would like to know if it is possible to use some commonly used inference acceleration frameworks such as VLLM and LMDEPLOY in the model loading section.

[JasperDekoninck]

Thanks! Unfortunately, we do not support specialized inference frameworks. While the core algortihm is not that difficult to reimplement, I know that at least for VLLM it would take quite some effort to work around their built-in way to run inferenxe to make it work :(