Commit b100615
authored
[Feature] change ngram spec deode to triton op (vllm-project#11925)
### What this PR does / why we need it?
refer to: vllm-project#9008 . In
order to reduce subsequent maintenance costs,change ngram_spec_decode
ascend c op to triton op. In long context, the effect of Triton operator
is basically the same as that of Ascend C operator
test results:
- input length 128/ output length 100: ttft +5.4%( 221-> 233), tpot +10%
(40 -> 44)
- input length 1024/ output length 100: ttft +5%(2620->2767), tpot
0%(67->69)
- input length 21000/output length 100: ttft +0.86%(186155->187761) ,
tpot 0%(219.79->220)
Collaborate with @CarterDuan
- vLLM version: v0.25.1
- vLLM main:
vllm-project/vllm@d02df74
---------
Signed-off-by: HF-001 <1670186653@qq.com>1 parent 87c1ceb commit b100615
17 files changed
Lines changed: 273 additions & 1789 deletions
File tree
- csrc
- attention/ngram_spec_decode
- op_host
- op_api
- op_kernel
- tests/e2e
- nightly/single_node/ops/singlecard_ops
- pull_request/one_card/spec_decode
- vllm_ascend
- ops/triton/spec_decode
- worker
This file was deleted.
Lines changed: 0 additions & 82 deletions
This file was deleted.
This file was deleted.
Lines changed: 0 additions & 72 deletions
This file was deleted.
Lines changed: 0 additions & 118 deletions
This file was deleted.
Lines changed: 0 additions & 26 deletions
This file was deleted.
0 commit comments