Skip to content

Feature/kvcache per token - #5

Open
JartX wants to merge 119 commits into
mainfrom
feature/kvcache_per_token
Open

Feature/kvcache per token#5
JartX wants to merge 119 commits into
mainfrom
feature/kvcache_per_token

Conversation

@JartX

@JartX JartX commented Mar 27, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Owner

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

JartX and others added 30 commits March 12, 2026 15:32
@JartX @EricccYang
int8 kv cache per-token per-head scale
5da77fe 
Signed-off-by: JartX <sagformas@epdcenter.es>
Co-authored-by: yangyang4991 <yangyang4991@gmail.com>
@JartX
pre-commit
6f830fd 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
pre-commit
230d37e 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
isolate flows
325f7b8 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
isolate flows
7b15ebc 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Merge branch 'main' into feature/kvcache_int8_scale_per_token_continu…
fab75a6 
…e_pr_34327_thanks_ericccyang

Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
changed warning message
c5a61f2 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Fix assertion for kv_cache_torch_dtype
503f7b8 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
working on kvcache int8
eb96acf 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
pre-commit
42e914e 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
working in the test
6411697 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
pre-commit
6dcbc44 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Merge branch 'main' into feature/kvcache_int8_scale_per_token_continu…
3d293cc 
…e_pr_34327_thanks_ericccyang
@JartX
Merge branch 'main' into feature/kvcache_int8_scale_per_token_continu…
1a2bd45 
…e_pr_34327_thanks_ericccyang
@JartX
NotImplementedError at detect has_kv_transfer_group
0e30d8a 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
pre-commit
d351be8 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Merge branch 'main' into feature/kvcache_int8_scale_per_token_continu…
70ff12c 
…e_pr_34327_thanks_ericccyang
@yewentao256 @Isotr0py
Apply suggestions from code review
9f16e7a 
Co-authored-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
@JartX
fix key_cache_shape with [:3]
573b6cd 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
post-dot dequantization
82f96cf 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
removed notimplementederror requested by @NickLucche
1e91fdc 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
test changes
af878f8 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
test changes
d935061 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Reserve GPU memory for INT8 per-token scale caches
781306e 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
changed doc string
446fae6 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
refactor flex
ca3f02f 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Generalize per-token KV cache scale infrastructure for extensibility
4ee2689 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
refactor to use kv_cache_uses_per_token_scales
9794165 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
torch.empty in ensure_per_token_scale_caches
a96abe5 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Merge branch 'main' into feature/kvcache_int8_scale_per_token_continu…
c6c8b8e 
…e_pr_34327_thanks_ericccyang
JartX added 13 commits March 28, 2026 14:54
@JartX
restore position of comment
31771ae 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
optimize memory band usage
8a90f51 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
optimize performance
027e07f 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
restore mask.to
f690e22 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
optimize decode
92814c5 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
improve maintainability of per-token KV cache quantization
fc2d78f 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
fix import
f6e979b 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
pre-commit
f717fea 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
maintainability
4fb42ce 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Merge branch 'main' into feature/kvcache_per_token
3ef0a17
@JartX
changes requested by tjtanaa
58fb69c 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
Consolidate per-token KV cache quantization tests
38c79dd 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
pre-commit
3824c33 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
JartX force-pushed the feature/kvcache_per_token branch from 45abb8c to 3824c33 Compare March 29, 2026 12:19
JartX added 4 commits March 30, 2026 10:46
@JartX
Merge branch 'main' into feature/kvcache_per_token
bf0e29b
@JartX
Merge branch 'main' into feature/kvcache_per_token
059d3ff
@JartX
Merge branch 'main' into feature/kvcache_per_token
a7003a1 
# Conflicts:
#	vllm/config/cache.py
#	vllm/model_executor/layers/quantization/kv_cache.py
#	vllm/v1/attention/backend.py
#	vllm/v1/attention/backends/triton_attn.py
@JartX
change to end per_token
9b3e5b4 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
JartX force-pushed the feature/kvcache_per_token branch from 18116fb to 9b3e5b4 Compare April 1, 2026 13:10
JartX added 4 commits April 1, 2026 15:13
@JartX
align with mgoin
e3ef588 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
align with mgoin add _unify_page_size_per_token
adcf2cd 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
add assert
8c26b94 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
precommit
e0895b4 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
JartX force-pushed the feature/kvcache_per_token branch from 1b5c40c to 17a6e53 Compare April 1, 2026 16:27
@JartX
after talk with mgoin We're going back to per token head, but keeping…
05aff60 
… the modifications required after the meetings

fix: include kv_quant_mode in hybrid block size alignment
fix: carve scale caches from raw KV allocation to avoid OOM

Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
JartX force-pushed the feature/kvcache_per_token branch from f18fda9 to 05aff60 Compare April 1, 2026 16:59
JartX added 2 commits April 1, 2026 19:00
@JartX
precommit
2b6581a 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
per_token_head_scales
40a6d21 
Signed-off-by: JartX <sagformas@epdcenter.es>
@JartX
JartX force-pushed the feature/kvcache_per_token branch from 8020c4b to 40a6d21 Compare April 1, 2026 21:54
@JartX
fix test on upstream
8028987 
Signed-off-by: JartX <sagformas@epdcenter.es>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@JartX @yewentao256