Retained Q4K materialization produces NaN in Gemma4 FFN down projection on Apple Metal

[clocksmith]

Summary

retainQ4KMaterialization: true produces NaN/Inf for Gemma4 E2B Q4K on Apple M3 Metal. The mitigation in 59c2fd1 disables retained Q4K materialization for this model family on Apple Metal, but that is a capability gate/workaround rather than a root fix.

Repro Context

• Hardware/backend: Apple M3 Metal
• Model: gemma-4-e2b-it-q4k-ehf16-af32
• Runtime profile: profiles/gemma4-e2b-throughput
• Trigger: force runtimeConfig.runtime.inference.session.retainQ4KMaterialization=true
• Command class: verify, maxTokens: 8

Observed

Before the gate, forced retained Q4K materialization tripped the finiteness guard during Gemma4 inference. Diagnostic tracing with recorder batching disabled localized the first traced NaN to:

• first NaN boundary: L0.ffn_down
• prior checked boundary: L0.pre_ffn_norm was finite

This points at the retained Q4K FFN down projection path on Apple Metal, not attention/final logits as the primary source.

Current Mitigation

Commit 59c2fd1 added a capability rule that rewrites Apple Metal Gemma4 E2B Q4K sessions away from retainQ4KMaterialization: true. It also enforces the rule at loader materialization so explicit runtime overrides cannot re-enable the unstable path on Apple Metal.

Default throughput and the forced-old-path repro both pass after the gate, but the underlying retained materialization bug remains.

Next Work

• Compare retained vs non-retained Q4K materialization for the Gemma4 ffn_down projection on Apple Metal.
• Inspect buffer layout/dtype assumptions at the retained materialization boundary.
• Add a focused per-op or per-layer regression for L0.ffn_down under retained Q4K.
• Fix the retained path and then remove or narrow the Apple Metal disable rule.