2 related paper on KV-cache management/compression

[RaghavvGoel]

Hi,

There are two more papers which I believe are super related :)

1. KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments (https://arxiv.org/abs/2504.15364)
2. CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction (https://arxiv.org/abs/2504.14051)

It would be great if you could consider adding them

[October2001]

Sure, thanks for the excellent work!

I've added it to this repository.

[jagmarques]

Both papers are good additions — thanks for surfacing them.

KeyDiff (key similarity-based eviction) fits into the attention-score-free eviction family alongside methods like LESS and SirLLM. The key insight — that similar keys can be merged or dropped — is related to but distinct from methods that use actual attention weights for eviction decisions. Worth categorizing separately in the list.

CAOTE's attention output error metric is interesting because it is trying to approximate the downstream effect of eviction rather than using a proxy like attention score magnitude. This is closer in spirit to calibration-based methods (like DP-calibrated quantization) where you measure impact on the output distribution rather than the cache representation itself.

A few other recent papers that might be worth adding if not already present:

• TurboQuant / PolarQuant (ICLR 2026): quantization via polar coordinate transformation, achieves strong results at low bit-widths
• PALU (ICLR 2025): low-rank projection of KV cache, complementary to quantization
• NexusQuant (https://github.com/jagmarques/nexusquant): lattice vector quantization (E8) combined with Hadamard rotation and temporal predictive coding — takes a different mathematical approach than scalar quantization or eviction

One organizational suggestion: it might help to split the list into eviction-based methods (which change which tokens are cached) vs. compression-based methods (which change how cached tokens are represented). They compose well but address different parts of the memory problem, and the literature tends to conflate them.

[October2001]

Thanks for your suggestions!
I've added these papers to the repo.

[sravanthkod]

Hi,

There are two more papers which I believe are super related :)

1. KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments (https://arxiv.org/abs/2504.15364)
2. CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction (https://arxiv.org/abs/2504.14051)

It would be great if you could consider adding them

Would be really helpful, if Keydiff & CAOTE codes are released.