Nvidia V100 speculative decoding slower then AR

[merbanan]

DFLASH27B_KV_TQ3=1 DFLASH_SAMPLED_VERIFY=1 LUCE_MMVQ_MAX_NCOLS=8 ./server/build/dflash_server server/models/Qwen3.6-27B-Q4_K_M.gguf --draft server/models/draft/dflash-draft-3.6-q4_k_m.gguf --ddtree --ddtree-budget 2 --port 8001 --max-ctx 200000

I need to use this command to get speculative decode to work at all. But this is still slower then AR mode on V100. What is needed to get the V100 speculative decoding working faster then AR?

[merbanan]

After some testing it seems like speculative decoding is actually working, at least according to the tests, Claude has this to say:

V100 baseline established (HumanEval):

• AR: 25.1 tok/s (throttles to ~19 under sustained load; cold-start ~28)
• DFlash: 53.3 tok/s, AL=6.42
• Speedup: 2.12× — spec decode is already beating AR

Key gaps vs RTX 3090 (3.43×, AL=8.31):

1. Thermal throttling cuts the AR floor by ~30% during sustained runs
2. Acceptance length is 23% lower (6.42 vs 8.31) — FP16 draft rounding is the suspected cause

Now interacting with the model through OpenCode works much better then before.

I do have some patches and it crashed 8/10 math tests so it is not really stable.

[davide221]

Thanks @merbanan. Feel free to share your patches to make V100 spec.decode go fast. Yes fp16 seems the most plausible culprit for this discrepancy in AL.