Skip to content

vendor: grouped-expert MUL_MAT_ID for small MoE verify batches (lagunaxs2.1 porting) #501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
davide221 merged 20 commits into from
Jul 10, 2026
+3,157 −326
Merged

vendor: grouped-expert MUL_MAT_ID for small MoE verify batches (lagunaxs2.1 porting) #501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
63cbb59
vendor: grouped-expert MUL_MAT_ID for small MoE verify batches
davide221 Jul 8, 2026
c5e61b5
vendor+server: opt-in per-token adaptive expert count for MoE verify …
davide221 Jul 8, 2026
1fe86bd
vendor+server: drafter-confidence adaptive verify width for MoE spec …
davide221 Jul 8, 2026
1238629
server: make validated MoE verify optimizations the default, single f…
davide221 Jul 8, 2026
4a4d519
fix(draft): load and apply Laguna XS 2.1 drafter tensors (attn gates,…
davide221 Jul 8, 2026
61b6abd
perf(spec): device-resident domino head, persistent draft graph, draf…
Jul 9, 2026
746a81b
feat(spec): drafter context-KV ring cache (target-agnostic)
davide221 Jul 10, 2026
b645e48
feat(laguna): flat long-context serving under KVFlash
davide221 Jul 10, 2026
1b5a5fb
feat(qwen35): adopt the drafter context-KV ring cache
davide221 Jul 10, 2026
bf38461
fix(spec): address review findings on the drafter ring cache
davide221 Jul 10, 2026
0942a2d
fix(review): harden the mmid/adaptive-K/ring surfaces for merge
davide221 Jul 10, 2026
8c4f08a
refactor(env): consolidate profilers into DFLASH_PROF, document the e…
davide221 Jul 10, 2026
4b85903
polish(review): log-spam guard, honest comments, doc rendering
davide221 Jul 10, 2026
5469578
fix(deploy): no tq3_0 or finite fa-window in any default path
davide221 Jul 10, 2026
9bfd119
feat(laguna): tool calling end to end
davide221 Jul 10, 2026
b71c23c
refactor(tools): hoist the duplicated trim lambda into trim_ws
davide221 Jul 10, 2026
a0704e5
fix(harness): purge finite fa-window and tq3_0 defaults everywhere
davide221 Jul 10, 2026
871bbf6
fix(tools): laguna dialect review findings
davide221 Jul 10, 2026
b3ac286
fix(server): status report no longer claims the removed tq3_0 auto-de…
davide221 Jul 10, 2026
a96edf2
docs(laguna): drop stale reference to the removed tq3 auto policy
davide221 Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions harness/benchmarks/run_lucebox_vs_llamacpp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,12 @@ MAX_CTX="${MAX_CTX:-32768}"
MAX_TOKENS="${MAX_TOKENS:-256}"
BUDGET="${BUDGET:-22}"
VERIFY_MODE="${VERIFY_MODE:-ddtree}"
FA_WINDOW="${FA_WINDOW:-2048}"
CACHE_TYPE_K="${CACHE_TYPE_K:-tq3_0}"
CACHE_TYPE_V="${CACHE_TYPE_V:-tq3_0}"
# 0 = full attention; finite windows break tool calls and are opt-in only.
FA_WINDOW="${FA_WINDOW:-0}"
# Empty = the server picks the model family's KV default (tq3_0/q4_0 garble
# laguna); set explicitly to experiment.
CACHE_TYPE_K="${CACHE_TYPE_K:-}"
CACHE_TYPE_V="${CACHE_TYPE_V:-}"
EXTRA_SERVER_ARGS="${EXTRA_SERVER_ARGS:---lazy-draft}"
LLAMA_N_GPU_LAYERS="${LLAMA_N_GPU_LAYERS:-999}"
MODEL_ID="${MODEL_ID:-luce-dflash}"
Expand Down Expand Up @@ -124,8 +127,8 @@ local_fa_args=()
if [[ -n "$FA_WINDOW" ]] && [[ "$FA_WINDOW" != "0" ]]; then
local_fa_args=(--fa-window "$FA_WINDOW")
fi
export DFLASH27B_KV_K="$CACHE_TYPE_K"
export DFLASH27B_KV_V="$CACHE_TYPE_V"
if [[ -n "$CACHE_TYPE_K" ]]; then export DFLASH27B_KV_K="$CACHE_TYPE_K"; fi

@cubic-dev-ai cubic-dev-ai Bot Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: When CACHE_TYPE_K/V are empty (new default), the server is intended to pick its own KV cache default. But if the parent shell already has DFLASH27B_KV_K exported (e.g. from a previous benchmark run or shell profile), that value leaks into the server's environment and silently overrides the server's default. Recommend unsetting the env var when the bash variable is empty.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At harness/benchmarks/run_lucebox_vs_llamacpp.sh, line 130:

<comment>When CACHE_TYPE_K/V are empty (new default), the server is intended to pick its own KV cache default. But if the parent shell already has DFLASH27B_KV_K exported (e.g. from a previous benchmark run or shell profile), that value leaks into the server's environment and silently overrides the server's default. Recommend unsetting the env var when the bash variable is empty.</comment>

<file context>
@@ -124,8 +127,8 @@ local_fa_args=()
 fi
-export DFLASH27B_KV_K="$CACHE_TYPE_K"
-export DFLASH27B_KV_V="$CACHE_TYPE_V"
+if [[ -n "$CACHE_TYPE_K" ]]; then export DFLASH27B_KV_K="$CACHE_TYPE_K"; fi
+if [[ -n "$CACHE_TYPE_V" ]]; then export DFLASH27B_KV_V="$CACHE_TYPE_V"; fi
 "$DFLASH_SERVER_BIN" "$TARGET" \
</file context>
Suggested change
if [[ -n "$CACHE_TYPE_K" ]]; then export DFLASH27B_KV_K="$CACHE_TYPE_K"; fi
if [[ -n "$CACHE_TYPE_K" ]]; then export DFLASH27B_KV_K="$CACHE_TYPE_K"; else unset DFLASH27B_KV_K; fi

if [[ -n "$CACHE_TYPE_V" ]]; then export DFLASH27B_KV_V="$CACHE_TYPE_V"; fi
"$DFLASH_SERVER_BIN" "$TARGET" \
--draft "$DRAFT" \
--host "$HOST" \
Expand Down
26 changes: 13 additions & 13 deletions harness/client_test_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ class ServerProfile:
"--budget", "22",
"--verify-mode", "ddtree",
"--max-ctx", "4096",
"--fa-window", "1024",
"--fa-window", "0",
"--cache-type-k", "q8_0",
"--cache-type-v", "q8_0",
"--prefix-cache-slots", "0",
Expand All @@ -140,9 +140,9 @@ class ServerProfile:
"--budget", "22",
"--verify-mode", "ddtree",
"--max-ctx", "8192",
"--fa-window", "2048",
"--cache-type-k", "tq3_0",
"--cache-type-v", "tq3_0",
"--fa-window", "0",
"--cache-type-k", "q8_0",
"--cache-type-v", "q8_0",
"--prefix-cache-slots", "0",
"--prefill-cache-slots", "0",
),
Expand All @@ -153,9 +153,9 @@ class ServerProfile:
"--budget", "22",
"--verify-mode", "ddtree",
"--max-ctx", "16384",
"--fa-window", "2048",
"--cache-type-k", "tq3_0",
"--cache-type-v", "tq3_0",
"--fa-window", "0",
"--cache-type-k", "q8_0",
"--cache-type-v", "q8_0",
"--prefix-cache-slots", "0",
"--prefill-cache-slots", "0",
),
Expand All @@ -166,9 +166,9 @@ class ServerProfile:
"--budget", "16",
"--verify-mode", "ddtree",
"--max-ctx", "32768",
"--fa-window", "2048",
"--cache-type-k", "tq3_0",
"--cache-type-v", "tq3_0",
"--fa-window", "0",
"--cache-type-k", "q8_0",
"--cache-type-v", "q8_0",
"--prefix-cache-slots", "0",
"--prefill-cache-slots", "0",
"--lazy-draft",
Expand All @@ -181,9 +181,9 @@ class ServerProfile:
"--budget", "16",
"--verify-mode", "ddtree",
"--max-ctx", "32768",
"--fa-window", "2048",
"--cache-type-k", "tq3_0",
"--cache-type-v", "tq3_0",
"--fa-window", "0",
"--cache-type-k", "q8_0",
"--cache-type-v", "q8_0",
"--prefix-cache-slots", "0",
"--prefill-cache-slots", "0",
"--prefill-compression", "auto",
Expand Down
22 changes: 14 additions & 8 deletions harness/clients/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,16 @@ LLAMA_UPSTREAM_PORT="${LLAMA_UPSTREAM_PORT:-$((PORT + 1))}"
MAX_CTX="${MAX_CTX:-16384}"
VERIFY_MODE="${VERIFY_MODE:-seq}"
BUDGET="${BUDGET:-22}"
FA_WINDOW="${FA_WINDOW:-2048}"
CACHE_TYPE_K="${CACHE_TYPE_K:-tq3_0}"
CACHE_TYPE_V="${CACHE_TYPE_V:-tq3_0}"
LLAMA_CACHE_TYPE_K="${LLAMA_CACHE_TYPE_K:-$CACHE_TYPE_K}"
LLAMA_CACHE_TYPE_V="${LLAMA_CACHE_TYPE_V:-$CACHE_TYPE_V}"
# FA_WINDOW: 0 = full attention. A finite window is known to break tool
# calling (Qwen3.6) - only set it explicitly for experiments.
FA_WINDOW="${FA_WINDOW:-0}"
# CACHE_TYPE_K/V: empty = let the server pick the model family's default
# (laguna q8_0, q4_0 base; forcing tq3_0/q4_0 on laguna garbles its
# output). Set explicitly to experiment.
CACHE_TYPE_K="${CACHE_TYPE_K:-}"
CACHE_TYPE_V="${CACHE_TYPE_V:-}"
LLAMA_CACHE_TYPE_K="${LLAMA_CACHE_TYPE_K:-${CACHE_TYPE_K:-q8_0}}"
LLAMA_CACHE_TYPE_V="${LLAMA_CACHE_TYPE_V:-${CACHE_TYPE_V:-q8_0}}"
MAX_TOKENS="${MAX_TOKENS:-2048}"
EXTRA_SERVER_ARGS="${EXTRA_SERVER_ARGS:-}"

Expand Down Expand Up @@ -155,9 +160,10 @@ start_dflash_native_server() {
if [[ -n "$FA_WINDOW" ]] && [[ "$FA_WINDOW" != "0" ]]; then
fa_args=(--fa-window "$FA_WINDOW")
fi
# Export KV cache type env vars for the C++ server to pick up.
export DFLASH27B_KV_K="$CACHE_TYPE_K"
export DFLASH27B_KV_V="$CACHE_TYPE_V"
# Export KV cache type env vars for the C++ server to pick up (only when
# explicitly requested: the per-axis envs override family defaults).
if [[ -n "$CACHE_TYPE_K" ]]; then export DFLASH27B_KV_K="$CACHE_TYPE_K"; fi
if [[ -n "$CACHE_TYPE_V" ]]; then export DFLASH27B_KV_V="$CACHE_TYPE_V"; fi
"$DFLASH_SERVER_BIN" "$TARGET" \
"${draft_args[@]}" \
--host "$HOST" \
Expand Down
2 changes: 1 addition & 1 deletion harness/clients/run_openclaw.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
: "${MAX_CTX:=204800}"
: "${BUDGET:=22}"
: "${VERIFY_MODE:=ddtree}"
: "${FA_WINDOW:=2048}"
# FA_WINDOW stays 0 (common.sh): finite windows break tool calls.
: "${EXTRA_SERVER_ARGS:=--lazy-draft}"
source "$SCRIPT_DIR/common.sh"

Expand Down
1 change: 1 addition & 0 deletions server/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,7 @@ add_library(dflash_common STATIC
src/common/dflash_draft_ipc_daemon.cpp
src/common/pflash_drafter_ipc.cpp
src/common/dflash_draft_graph.cpp
src/common/dflash_draft_kv.cpp
src/common/dflash_spec_decode.cpp
src/common/layer_split_backend.cpp
src/common/layer_split_runtime.cpp
Expand Down
8 changes: 8 additions & 0 deletions server/deps/llama.cpp/ggml/include/ggml-cuda.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,14 @@ GGML_BACKEND_API void ggml_backend_cuda_unregister_host_buffer(void * buffer);

Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
GGML_BACKEND_API ggml_backend_reg_t ggml_backend_cuda_reg(void);

// [TAG_TOPK_ROWS] top-k (k <= 8) entries + softmax probabilities per row of a

@cubic-dev-ai cubic-dev-ai Bot Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The [TAG_TOPK_ROWS] prefix in the comment is inconsistent with every other function comment in this header, which uses plain descriptive text without bracket tags. Consider removing the tag or using it uniformly across the API surface to keep the documentation consistent.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At server/deps/llama.cpp/ggml/include/ggml-cuda.h, line 48:

<comment>The `[TAG_TOPK_ROWS]` prefix in the comment is inconsistent with every other function comment in this header, which uses plain descriptive text without bracket tags. Consider removing the tag or using it uniformly across the API surface to keep the documentation consistent.</comment>

<file context>
@@ -45,6 +45,12 @@ GGML_BACKEND_API void ggml_backend_cuda_unregister_host_buffer(void * buffer);
 
 GGML_BACKEND_API ggml_backend_reg_t ggml_backend_cuda_reg(void);
 
+// [TAG_TOPK_ROWS] top-k (k <= 8) entries + softmax probabilities per row of a
+// device-resident contiguous F32 [ncols, nrows] tensor; ~KB of readback. Not
+// a graph op: call after the producing graph_compute has returned.
</file context>
Suggested change
// [TAG_TOPK_ROWS] top-k (k <= 8) entries + softmax probabilities per row of a
// top-k (k <= 8) entries + softmax probabilities per row of a

// device-resident contiguous F32 [ncols, nrows] tensor. probs_out and ids_out
// must each hold k * nrows elements (row-major: entry [r*k + j] = rank-j of
// row r). Not a graph op: call only after the SYNCHRONOUS
// ggml_backend_graph_compute() producing `logits` has returned.
GGML_BACKEND_API bool ggml_backend_cuda_topk_rows(const struct ggml_tensor * logits, int k,
float * probs_out, int32_t * ids_out);

#ifdef __cplusplus
}
#endif
15 changes: 14 additions & 1 deletion server/deps/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3265,6 +3265,14 @@ static bool ggml_cuda_graph_update_required(ggml_backend_cuda_context * cuda_ctx
}

if (res || memcmp(&graph->node_props[i], &prop, sizeof(prop)) != 0) {
// [TAG_FUSED_LOOP] with GGML_CUDA_GRAPH_STATS, name the first node
// whose properties changed — the reason a graph re-captures.
static const bool dbg = getenv("GGML_CUDA_GRAPH_STATS") != nullptr;
if (dbg && !res) {
GGML_LOG_INFO("[graph-mismatch] key=%p node=%d op=%s name=%s\n",
graph_key, i, ggml_op_name(cgraph->nodes[i]->op),
cgraph->nodes[i]->name);
}
graph->node_props[i] = prop;
res = true;
}
Expand Down Expand Up @@ -4380,7 +4388,12 @@ static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend,
if (use_cuda_graph && !cuda_graph_update_required) g->stat_replay++;
else if (use_cuda_graph) g->stat_capture++;
else g->stat_eager++;
if (g->stat_total % 200 == 0) {
static const int stats_every = [](){
const char * e = getenv("GGML_CUDA_GRAPH_STATS_EVERY");
const int v = e ? atoi(e) : 200;
return v > 0 ? v : 200; // guard % 0 on malformed input
}();
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
if (g->stat_total % stats_every == 0) {
GGML_LOG_INFO("[cuda-graph-stats] key=%p n_nodes=%d total=%llu replay=%llu capture=%llu eager=%llu enabled=%d\n",
graph_key, cgraph->n_nodes,
(unsigned long long)g->stat_total, (unsigned long long)g->stat_replay,
Expand Down
Loading
Loading