feat: KV cache quantization - #47

Open

Skyrion9 wants to merge 6 commits into

rodrigomatta:mainfrom

Skyrion9:KV-quant

Commits on Jul 19, 2026

feat: mmap-based lazy weight loading implementation
Skyrion9
committed
ad61fbb
feat: phase-gated VRAM swap and hot-swap state machine implementation for server mode
Skyrion9
committed
c6680ad

Commits on Jul 22, 2026

perf: use sequential access hints for weight loading
Skyrion9
committed
8bca5de

Commits on Jul 30, 2026

feat: more VRAM split (codec), deferred init, and overlapped decode pipeline
Skyrion9
committed
92e48f0

Commits on Aug 3, 2026

feat: KV cache reuse and prompt prefill caching
Skyrion9
committed
0dc1aee

Commits on Aug 5, 2026

feat: KV cache quantization user params (--cache-type-k / --cache-type-v)
Skyrion9
committed
e01cc73