Skip to content

Fix Gemma 4 default prefill and prepare 0.2.5 - #103

Merged
bsbodden merged 6 commits into
mainfrom
agent/default-qualification-gate
Aug 3, 2026
Merged

Fix Gemma 4 default prefill and prepare 0.2.5#103
bsbodden merged 6 commits into
mainfrom
agent/default-qualification-gate

Conversation

@bsbodden

@bsbodden bsbodden commented Aug 3, 2026

Copy link
Copy Markdown
Member

What changed

  • fix singleton Gemma 4 routed-expert prefill projections by copying capacity-sized batched buffers through reusable exact-shape scratch arrays
  • run a complete one-iteration RAG correctness smoke with public Models defaults before applying performance tuning
  • require every workload attempt to succeed with deterministic correctness, abstention, and prefix-cache reuse
  • retain nested benchmark failures and immutable default-smoke evidence
  • prepare the coordinated Models 0.2.5 patch release metadata

Root cause

The 0.2.4 singleton fallback correctly left the native batch kernel, but then passed a capacity-sized prefill backing array to the strict one-vector Java projection. On the tester's M1, a one-token 2,816-column expert projection therefore arrived as 360,448 elements (128 × 2,816) and failed dimension validation.

The corrected fallback reuses scratch arrays sized to the exact matrix input and output shape, copies one row into them, performs the projection, and copies the result back into the batched output buffer.

User impact

The qualified ggml-org/gemma-4-26B-A4B-it-GGUF Q4_K_M artifact now prefills through ModelJars' default Rust/FFM backend without either singleton failure:

  • loaded Gemma 4 batched-prefill kernel rejected Q4_K projection ...
  • Vector dimensions differ: 360448 != 2816

Validation

  • ./gradlew --no-daemon spotlessCheck verifyReleaseMetadata :backend-java:test --tests com.integrallis.models.backend.purejava.gemma4.Gemma4ForwardPassTest :models-rag-bench:test --tests com.integrallis.models.rag.ControlledRagQualificationScriptTest --tests com.integrallis.models.rag.CertifiedRagEvidenceTest
  • full GitHub CI, native-kernel, CodeQL, large-model, and model-integration checks on the fixed runtime
  • exact Gemma artifact SHA-256 88f4a13b0bb95f031a7fad973e10854122fb67ebc34d214d39a2f65053046abc
  • physical Apple Silicon qualification with JDK 25.0.4, 128-bit Vector API, rust-ffm, and gemma4-batched-prefill enabled:
Hardware RAM Focused reproducer Full Gemma matrix Abstention Runtime failures
Apple M1 16 GiB 1/1 9/9 correct 0
Apple M2 Pro 32 GiB 1/1 9/9 correct 0
Apple M4 Pro 48 GiB 1/1 9/9 correct 0

M3 remains pending provider quota and is not claimed by this PR.

@bsbodden bsbodden changed the title Gate qualification on library defaults Fix Gemma 4 default prefill and prepare 0.2.5 Aug 3, 2026
@bsbodden
bsbodden marked this pull request as ready for review August 3, 2026 17:54
@bsbodden
bsbodden merged commit 3d31253 into main Aug 3, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant