Skip to content

Commit b7b7f4e

Browse files
authored
[BugFix]Block count calculation error in MambaManager when hitting the external cache (vllm-project#11129)
### What this PR does / why we need it? - When external cache hit, MambaManager will allocates 1 block for external cache + 1 block for new scheduled tokens, But upstream MambaManager#get_num_blocks_to_allocate says it only needs one extra block besides the speculative blocks, which isn't right, this pr overrides the get_num_blocks_to_allocate method to return one more block for this condition - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: Qingsong Zhang <1640410765@qq.com>
1 parent dd178e8 commit b7b7f4e

2 files changed

Lines changed: 45 additions & 6 deletions

File tree

vllm_ascend/patch/__init__.py

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -951,12 +951,18 @@
951951
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
952952
# 1. `vllm.v1.core.single_type_kv_cache_manager.MambaManager`
953953
# Why:
954-
# Upstream hybrid prefix cache lookup does not support PCP/DCP.
954+
# 1. Upstream hybrid prefix cache lookup does not support PCP/DCP.
955+
# 2. Upstream MambaManager#get_num_blocks_to_allocate give the
956+
# wrong number of blocks when an external cache hit occurred
955957
# How:
956-
# Replace MambaManager with AscendMambaManager for prefix cache hit lookup
957-
# on hybrid Mamba paths (logical mamba block_size when caching is enabled).
958+
# 1. Replace MambaManager with AscendMambaManager for prefix cache hit lookup
959+
# on hybrid Mamba paths (logical mamba block_size when caching is enabled).
960+
# 2. Override the get_num_blocks_to_allocate method to fix the number of blocks
961+
# when hitting the external cache and loading synchronously
958962
# Related PR (if no, explain why):
959-
# https://github.com/vllm-project/vllm/pull/40996
963+
# 1. https://github.com/vllm-project/vllm/pull/40996
964+
# 2. https://github.com/vllm-project/vllm/pull/46892
960965
# Future Plan:
961-
# Upstream PR #40996 adds hybrid prefix cache lookup for DCP only; PCP is
962-
# not supported yet. Remove this patch once upstream supports both PCP and DCP.
966+
# 1. Upstream PR #40996 adds hybrid prefix cache lookup for DCP only; PCP is
967+
# not supported yet. Remove this patch once upstream supports both PCP and DCP.
968+
# 2. Remove this patch once upstream accept 46892 pr or fixed the bug by other pr.

vllm_ascend/patch/platform/patch_mamba_manager.py

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
55
# This file is a part of the vllm-ascend project.
66
#
7+
from collections.abc import Sequence
8+
79
import vllm.v1.core.single_type_kv_cache_manager as single_type_kv_cache_manager
810
from vllm.v1.core.single_type_kv_cache_manager import (
911
BlockHashList,
@@ -47,5 +49,36 @@ def find_longest_cache_hit(
4749
break
4850
return computed_blocks
4951

52+
def get_num_blocks_to_allocate(
53+
self,
54+
request_id: str,
55+
num_tokens: int,
56+
new_computed_blocks: Sequence[KVCacheBlock],
57+
total_computed_tokens: int,
58+
num_tokens_main_model: int,
59+
apply_admission_cap: bool = False,
60+
) -> int:
61+
num_new_blocks = super().get_num_blocks_to_allocate(
62+
request_id,
63+
num_tokens,
64+
new_computed_blocks,
65+
total_computed_tokens,
66+
num_tokens_main_model,
67+
apply_admission_cap,
68+
)
69+
# When external KV cache is loaded synchronously with new
70+
# tokens, allocate_new_computed_blocks() allocates one
71+
# extra block to hold the external cache content. Account
72+
# for it here so the free-capacity check is accurate.
73+
# (External tokens exist when total_computed_tokens exceeds
74+
# what local prefix-cache hits cover; sync loading when
75+
# num_tokens_main_model exceeds total_computed_tokens.)
76+
has_external_tokens = total_computed_tokens > len(new_computed_blocks) * self.block_size
77+
has_new_scheduled_tokens = num_tokens_main_model > total_computed_tokens
78+
if has_external_tokens and has_new_scheduled_tokens:
79+
# one more block for external computed tokens
80+
num_new_blocks += 1
81+
return num_new_blocks
82+
5083

5184
single_type_kv_cache_manager.MambaManager = AscendMambaManager

0 commit comments

Comments
 (0)