fix(dflash): scale DeepSeek4 graph sizing for large chunks

[weicj]

Summary

This PR follows on #489 , which continues the current DeepSeek4 mainline from #353 and the direct pure-UMA runtime route, and scales DeepSeek4 graph and metadata sizing for large prefill chunks so --chunk behaves as a real throughput knob instead of hitting small-graph limits from the older defaults.

In the current runtime, larger chunk sizes could fail in three different ways:

• full-step graph node capacity too small;
• full-step ggml metadata arena too small;
• attention-step graph / metadata sizing too small in the HC layer-range path.

This PR makes those capacities token-count aware. It does not change backend selection, placement policy, or execution semantics.

Changes

• Add token-count-aware full-step graph sizing.
• Add token-count-aware full-step metadata arena sizing.
• Add token-count-aware attention-step graph sizing.
• Add token-count-aware attention-step metadata sizing.
• Leave decode behavior and placement behavior unchanged.

Validation

• Runtime validation in a pure UMA gfx1151 environment with max_ctx=1048576

Native 1M ctx runtime checks:

4096 / 128

Chunk	Prefill tok/s	Decode tok/s	E2E walltime
512	222.26	15.40	26.7 s
1024	230.74	15.45	26.0 s
2048	241.02	15.50	25.3 s
4096	262.93	15.50	23.8 s

8192 / 256

Chunk	Prefill tok/s	Decode tok/s	E2E walltime
512	192.69	14.70	59.9 s
1024	200.30	14.70	58.3 s
2048	208.73	14.70	56.7 s
4096	224.38	14.60	54.0 s

16384 / 512

Chunk	Prefill tok/s	Decode tok/s	E2E walltime
512	150.62	13.30	147.2 s
1024	154.56	13.27	144.6 s
2048	162.71	13.30	139.2 s
4096	172.60	13.30	133.5 s

These checks show the prefill side scaling upward as chunk size increases, while decode remains effectively flat.

[cubic-dev-ai]

2 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_graph.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_graph.cpp:2794">
P2: `full_graph_build_us` telemetry only measures `ggml_init`, not the full graph build. The timing starts right before `ggml_init(params)` and ends immediately after it, missing the actual graph construction work — creating all layer tensors (RMS norms, MLA attention, MoE FFNs, output head), calling `ggml_new_graph_custom`, and `ggml_build_forward_expand(gf, logits)`. The reported value will be a few microseconds per step (just context allocation), while graph construction is the real cost. Either extend the timing window to end after `ggml_build_forward_expand` (before `full_alloc_t0`), or rename the field to `full_graph_init_us` to match what it actually measures.</violation>
</file>

<file name="server/src/common/backend_factory.cpp">

<violation number="1" location="server/src/common/backend_factory.cpp:218">
P0: The new code unconditionally routes non-layer-split deepseek4 requests to DeepSeek4Backend (line +235), dropping the old fallback-to-LayerSplitAdapter path when remote_target_shard is enabled without layer splitting. DeepSeek4Backend has no remote_target_shard support (zero references in its codebase), so a request with --remote-target-bip-bin set but only one GPU will silently ignore the remote shard configuration. This breaks the remote-HIP-target flow that used to work through the adapter.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P0: The new code unconditionally routes non-layer-split deepseek4 requests to DeepSeek4Backend (line +235), dropping the old fallback-to-LayerSplitAdapter path when remote_target_shard is enabled without layer splitting. DeepSeek4Backend has no remote_target_shard support (zero references in its codebase), so a request with --remote-target-bip-bin set but only one GPU will silently ignore the remote shard configuration. This breaks the remote-HIP-target flow that used to work through the adapter.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/backend_factory.cpp, line 218:

<comment>The new code unconditionally routes non-layer-split deepseek4 requests to DeepSeek4Backend (line +235), dropping the old fallback-to-LayerSplitAdapter path when remote_target_shard is enabled without layer splitting. DeepSeek4Backend has no remote_target_shard support (zero references in its codebase), so a request with --remote-target-bip-bin set but only one GPU will silently ignore the remote shard configuration. This breaks the remote-HIP-target flow that used to work through the adapter.</comment>

<file context>
@@ -215,44 +215,33 @@ std::unique_ptr<ModelBackend> create_backend(const BackendArgs & args) {
-            cfg.chunk      = args.chunk;
-
-            auto backend = std::make_unique<DeepSeek4Backend>(cfg);
+        if (args.device.is_layer_split()) {
+            DeepSeek4LayerSplitAdapterConfig cfg;
+            cfg.target_path         = args.model_path;
</file context>

[cubic-dev-ai]

P2: full_graph_build_us telemetry only measures ggml_init, not the full graph build. The timing starts right before ggml_init(params) and ends immediately after it, missing the actual graph construction work — creating all layer tensors (RMS norms, MLA attention, MoE FFNs, output head), calling ggml_new_graph_custom, and ggml_build_forward_expand(gf, logits). The reported value will be a few microseconds per step (just context allocation), while graph construction is the real cost. Either extend the timing window to end after ggml_build_forward_expand (before full_alloc_t0), or rename the field to full_graph_init_us to match what it actually measures.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_graph.cpp, line 2794:

<comment>`full_graph_build_us` telemetry only measures `ggml_init`, not the full graph build. The timing starts right before `ggml_init(params)` and ends immediately after it, missing the actual graph construction work — creating all layer tensors (RMS norms, MLA attention, MoE FFNs, output head), calling `ggml_new_graph_custom`, and `ggml_build_forward_expand(gf, logits)`. The reported value will be a few microseconds per step (just context allocation), while graph construction is the real cost. Either extend the timing window to end after `ggml_build_forward_expand` (before `full_alloc_t0`), or rename the field to `full_graph_init_us` to match what it actually measures.</comment>

<file context>
@@ -2678,23 +2766,71 @@ bool deepseek4_step(
+    params.mem_size = cached_sg ? full_step_cache.meta_arena.size() : ctx_size;
+    params.mem_buffer = cached_sg ? full_step_cache.meta_arena.data() : nullptr;
     params.no_alloc = true;
+    const auto full_build_t0 = Ds4TimingClock::now();
     ggml_context * ctx = ggml_init(params);
     if (!ctx) return false;
</file context>