Skip to content

Commit b100615

Browse files
authored
[Feature] change ngram spec deode to triton op (vllm-project#11925)
### What this PR does / why we need it? refer to: vllm-project#9008 . In order to reduce subsequent maintenance costs,change ngram_spec_decode ascend c op to triton op. In long context, the effect of Triton operator is basically the same as that of Ascend C operator test results: - input length 128/ output length 100: ttft +5.4%( 221-> 233), tpot +10% (40 -> 44) - input length 1024/ output length 100: ttft +5%(2620->2767), tpot 0%(67->69) - input length 21000/output length 100: ttft +0.86%(186155->187761) , tpot 0%(219.79->220) Collaborate with @CarterDuan - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@d02df74 --------- Signed-off-by: HF-001 <1670186653@qq.com>
1 parent 87c1ceb commit b100615

17 files changed

Lines changed: 273 additions & 1789 deletions

File tree

csrc/attention/ngram_spec_decode/CMakeLists.txt

Lines changed: 0 additions & 19 deletions
This file was deleted.

csrc/attention/ngram_spec_decode/ngram_spec_decode_torch_adpt.h

Lines changed: 0 additions & 82 deletions
This file was deleted.

csrc/attention/ngram_spec_decode/op_host/CMakeLists.txt

Lines changed: 0 additions & 21 deletions
This file was deleted.

csrc/attention/ngram_spec_decode/op_host/ngram_spec_decode_def.cpp

Lines changed: 0 additions & 72 deletions
This file was deleted.

csrc/attention/ngram_spec_decode/op_host/ngram_spec_decode_tiling.cpp

Lines changed: 0 additions & 118 deletions
This file was deleted.

csrc/attention/ngram_spec_decode/op_host/ngram_spec_decode_tiling.h

Lines changed: 0 additions & 26 deletions
This file was deleted.

0 commit comments

Comments
 (0)