Roadmap and recommended integration path for OSCAR with speculative decoding

[ytgitfast]

Hi, thanks for open-sourcing OSCAR. The SGLang-based INT2 KV serving path is very interesting, especially the mixed-KV design with BF16 sink/recent tokens and INT2 long-history KV cache.

I would like to ask whether OSCAR is planned to support speculative decoding in SGLang, such as draft-model speculative decoding or EAGLE/DFLASH/MTP style speculative decoding.

From the current serving example, OSCAR uses options such as:

SGLANG_ENABLE_MIXED_KV_WINDOWS=1
SGLANG_OSCAR_K_ROTATION_PATH=...
SGLANG_OSCAR_V_ROTATION_PATH=...
SGLANG_MIXED_KV_PREFIX_TOKENS=64
SGLANG_MIXED_KV_RECENT_TOKENS=256
--kv-cache-dtype int2
--kv-cache-quant-group-size 128
--decode-attention-backend triton

I have a few questions about the intended roadmap and correct integration design:

1. Is OSCAR + speculative decoding currently supported, or is there a planned timeline for this feature?

2. For speculative decoding, what is the recommended KV-cache lifecycle?
   My current understanding is that speculative candidate tokens should probably stay in a temporary high-precision or extra-buffer region before acceptance. Only accepted tokens should be committed into the normal paged KV cache, enter the BF16 recent window first, and later be demoted into OSCAR INT2 history. Rejected candidate KV should be discarded and should not enter prefix cache or INT2 history. Is this the intended direction?

3. For a two-model speculative setup, should OSCAR be applied only to the target model KV cache at first, while the draft model keeps its own normal KV cache? Or do you expect both target and draft KV caches to use OSCAR?

4. For EAGLE-style speculation, are there additional constraints because the draft path is more closely tied to the target model internals?

5. How should SGLANG_MIXED_KV_RECENT_TOKENS interact with unverified speculative tokens? Should speculative tokens count as recent tokens before acceptance, or should they live in a separate temporary region and only enter the mixed-KV window after being accepted?

6. Are there any known issues or recommended design choices for combining OSCAR mixed-KV windows with SGLang speculative decoding extra token slots / lookahead slots?

A possible staged integration plan I am considering is:

• First verify OSCAR INT2 KV serving on the target model without speculation.
• Then enable speculative decoding while keeping unverified speculative KV in a separate temporary BF16 buffer.
• On acceptance, commit only accepted target-model KV into the normal OSCAR mixed-KV cache.
• On rejection, immediately release rejected KV and prevent it from entering prefix cache, page allocation statistics, or INT2 demotion.
• Finally benchmark acceptance rate, memory usage, and decode throughput against BF16 KV + speculative decoding.

Does this match how the OSCAR team would approach the integration?

Any guidance on the expected timeline and recommended implementation direction would be very helpful. OSCAR + speculative decoding seems like a promising combination, since OSCAR reduces long-context KV memory pressure while speculative decoding reduces target-model decoding steps. The main uncertainty is how to correctly handle speculative KV commit/rollback together with OSCAR’s mixed BF16/INT2 cache layout.

[huxiang088]

+1, Ping @zzz0906

[zzz0906]

Hi, thanks for pinging me and for the detailed analysis.

1. Currently OSCAR + speculative decoding is not supported yet, but I agree this is a feasible and valuable direction. Your proposed lifecycle makes sense to me.
2. For rotation, the speculator shall follow the same rules as OSCAR did to generate rotation.
3. Speculative KV should follow the OSCAR BF16 + INT2 window design, with only accepted tokens committed as final KV, and recent-token accounting should only advance based on accepted tokens (which means the actual max recent tokens = recent tokens + max lookahead).
4. For EAGLE-style speculative decoding, I also think this should fit OSCAR well, since the target-side long-context KV is exactly where OSCAR can provide the most benefit. We can start with target-side OSCAR integration first and keep the provisional/speculative KV safe in BF16 before commit.
5. I will discuss this with the team and try to push the implementation forward as soon as possible. I’ll also tune and evaluate the speculator acceptance length to see whether the combined setup gives a meaningful performance gain.

Thanks again for the integration proposal — this sounds very promising to me.

[ytgitfast]

Hi, thanks for pinging me and for the detailed analysis.

1. Currently OSCAR + speculative decoding is not supported yet, but I agree this is a feasible and valuable direction. Your proposed lifecycle makes sense to me.
2. For rotation, the speculator shall follow the same rules as OSCAR did to generate rotation.
3. Speculative KV should follow the OSCAR BF16 + INT2 window design, with only accepted tokens committed as final KV, and recent-token accounting should only advance based on accepted tokens (which means the actual max recent tokens = recent tokens + max lookahead).
4. For EAGLE-style speculative decoding, I also think this should fit OSCAR well, since the target-side long-context KV is exactly where OSCAR can provide the most benefit. We can start with target-side OSCAR integration first and keep the provisional/speculative KV safe in BF16 before commit.
5. I will discuss this with the team and try to push the implementation forward as soon as possible. I’ll also tune and evaluate the speculator acceptance length to see whether the combined setup gives a meaningful performance gain.

Thanks again for the integration proposal — this sounds very promising to me.

Thanks for the detailed reply!
Glad to hear this direction is feasible. No pressure from my side — just wanted to raise the use case.
Looking forward to future updates. Thanks again!