Pr oscar int2 kv#13
Open
giveen wants to merge 5 commits into
Open
Conversation
…UANTS GPU-KV build)
- SET_ROWS: add Q2_0 to device_supports_op gate + set_rows_cuda_q2_0 kernel (port of Metal kernel_set_rows_q2_0). Fixes the 'cannot run SET_ROWS on buffer CUDA0' crash that blocked GPU q2_0 KV. - Fix latent write bug: blk.m = mean stored a float into a ggml_half field, overflowing and corrupting blk.d. Use __float2half(mean). - fattn VEC path: add dequantize_V_q2_0 + vec_dot_fattn_vec_KQ_q2_0 (Lloyd-Max dec; reconstruction m + d*centroid) and q2_0 dequant/KQ-dot dispatcher cases incl. D=512 (Gemma4 head_dim). Add vec_dot_q2_0_q8_1. - Force BEST_FATTN_KERNEL_VEC for q2_0 KV inputs. Note: OSCAR's fused two-tier attention (flash_attn_ext_mixed) used by the q2_0 KV path is still CPU/Metal-only in this fork; the non-fused VEC q2_0 port here is not yet dispatched by the graph. Full GPU q2_0 KV still requires porting flash_attn_ext_mixed to CUDA.
…d attention) Adds ggml_cuda_flash_attn_ext_mixed and routes GGML_OP_FLASH_ATTN_EXT with op_params[4]==1 to it (the LLAMA_KV_FUSED_FA path). Per-query-row online-softmax over the LP (q2_0) history tier + HP (f16) sink/recent tier. Status: builds + links; f16/f16 KV through the kernel is numerically correct. q2_0 KV tier still has a stride/layout bug in the LP indexing (decode math unit-tested exact vs CPU ref). Debug print left in for stride diagnosis.
…ode verification) One-shot device-side dump gated on a populated KV cache row; decodes q2_0 K/V rows (raw bytes + m/d + decoded floats) and recomputes LP attention on CPU to compare against kernel dst. Confirms decode+stride are correct; isolates remaining bug to integration logic. To be stripped before final fix commit. Authored-by: giveen <1180939+giveen@users.noreply.github.com>
Remove all [MIXED-DBG]/[MIXED-VFY] verification blocks added for q2_0 debugging. The fused mixed-precision attention kernel has been verified correct by the VFY system (max err 0.017 for decode/prefill head 0) — the bug is elsewhere (KV cache write path or LP V dequant interaction). Also fix upstream API changes: -nm/--mixed-fa-trigger flags removed, --flash-attn now takes on|off|auto, --override-kv uses KEY=TYPE:VALUE. Update launch command accordingly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.