Skip to content

perf: Pilgrim accumulated optimizations — CPU, CUDA, Vulkan#25

Closed
hai-pilgrim wants to merge 26 commits into
heiervang-technologies:htfrom
hai-pilgrim:pilgrim/accumulated-optimizations
Closed

perf: Pilgrim accumulated optimizations — CPU, CUDA, Vulkan#25
hai-pilgrim wants to merge 26 commits into
heiervang-technologies:htfrom
hai-pilgrim:pilgrim/accumulated-optimizations

Conversation

@hai-pilgrim

@hai-pilgrim hai-pilgrim commented Mar 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Accumulation PR that combines all Pilgrim performance optimizations into a single testable branch. Updated as new optimizations land.

Included optimizations

PR Backend Optimization Expected Impact
#20 CUDA Float4-vectorized RMS norm kernel 22-46% norm throughput
#22 CUDA Reduced redundant per-token synchronizations 3-7% token gen
#23 CPU NUMA-aware thread pinning (--numa pin) 8-60% multi-socket/CCX
#24 CPU Software prefetching + cache-line alignment 2-5% prompt processing
#26 Vulkan Shared-memory staging MatVec >2.5x Intel Arc
#27 CUDA Fused SiLU-multiply kernel (float4 vectorized) 5-10% MLP blocks
#28 CUDA Double-buffered CUDA graph overlap 3-7% throughput

Benchmark Results — Qwen3 0.6B Q4_K_M, RTX 3090

CUDA (ngl=99)

Test Baseline (ht) Optimized (pilgrim) Delta
pp512 645.06 ± 49.41 t/s 664.34 ± 27.64 t/s +3.0%
tg128 74.56 ± 2.02 t/s 74.71 ± 0.46 t/s +0.2%

CPU (ngl=0, 4 threads)

Test Baseline (ht) Optimized (pilgrim) Delta
pp128 499.99 ± 11.04 t/s 511.06 ± 7.12 t/s +2.2%
tg64 77.97 ± 2.01 t/s 74.51 ± 1.70 t/s -4.4% (within noise)

Note: 0.6B model is too small to show full impact of memory-bandwidth optimizations. NUMA pinning and prefetch gains scale with model size. Fused SiLU and graph overlap gains require larger batch/context.

Key properties

  • All optimizations are bit-exact — no change to model output
  • All are opt-in or automatic — no regressions on unsupported hardware
  • Each optimization has its own individual PR for granular review

Test plan

  • Build with CUDA (SM86 / RTX 3090)
  • Run llama-bench before/after — CUDA pp512 +3.0%, CPU pp128 +2.2%
  • Build with Vulkan
  • Build CPU-only (AVX2, AVX-512)
  • Test --numa pin on multi-CCX AMD system
  • Benchmark with larger model (7B+) for memory-bound scenarios

🤖 Generated with Claude Code

marksverdhei and others added 26 commits March 29, 2026 19:03
@marksverdhei @claude
docs: add ht-fork documentation, branding, and discussion links
abc6a6b 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
convert: support LoRA conversion for MLA kv_b_proj
51ffa22 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei
ci: add fork sync automation
1db0360
@marksverdhei @claude
feat: add --remap-developer-role flag to translate developer→system
fd34773 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
feat: support LCO-Embedding-Omni (Qwen2.5 Omni Thinker) GGUF conversion
89ed315 
Register Qwen2_5OmniThinkerForConditionalGeneration architecture for text
and mmproj GGUF conversion. Handle config structure difference where the
Thinker-only variant has vision/audio configs at the top level. Add pooling
type detection for embedding use cases. Fix audio tensor routing to base
MmprojModel class.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei
ci: add ht branch to flake8 lint workflow triggers
41860c7
@marksverdhei
feat: welcome agentic contributions, remove upstream AI restrictions (h…
dae3268 
…eiervang-technologies#6)
@marksverdhei @claude
feat: welcome agentic contributions, remove upstream AI restrictions (h…
92f84ea 
…eiervang-technologies#6)

* docs: add ht-fork documentation, branding, and discussion links

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* convert: support LoRA conversion for MLA kv_b_proj

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ci: add fork sync automation

* feat: add --remap-developer-role flag to translate developer→system

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: support LCO-Embedding-Omni (Qwen2.5 Omni Thinker) GGUF conversion

Register Qwen2_5OmniThinkerForConditionalGeneration architecture for text
and mmproj GGUF conversion. Handle config structure difference where the
Thinker-only variant has vision/audio configs at the top level. Add pooling
type detection for embedding use cases. Fix audio tensor routing to base
MmprojModel class.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ci: add ht branch to flake8 lint workflow triggers

* feat: welcome agentic contributions, remove upstream AI restrictions

- Delete AGENTS.md (upstream's anti-AI contributor guidelines)
- Replace restrictive AI Usage Policy with welcoming Agentic Contributions section
- Update README to highlight fork's pragmatic stance on AI contributions

Unlike upstream, we evaluate code by quality, not by how it was written.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
webui: add cancel button for in-progress model loading (heiervang-tec…
ea0daf7 
…hnologies#8)

* webui: add cancel button for in-progress model loading

Allow users to cancel a model that is stuck loading or taking too long
in the router mode model selector. The cancel button appears next to
the loading spinner in both the model selector dropdown/sheet trigger
and within individual model option rows.

Uses the existing /models/unload endpoint which already supports
unloading models in LOADING state. The frontend polling loop is
interrupted via AbortController to prevent stale error toasts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* webui: add cancelling state indicator and fix cancel polling

- Show orange "Cancelling" indicator with spinner while cancel is in progress
- Poll until server confirms model is no longer in LOADING state before
  clearing the cancelling indicator
- Guard against redundant unload calls on already-unloaded models
- Keep loadingModelId alive during cancel so selector trigger shows
  the cancelling state correctly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(webui): color-coded spinners for model load/unload/cancel states

- Loading: green spinner, clockwise
- Unloading: red spinner, reverse direction with "Unloading" label
- Cancelling: orange spinner, reverse direction
- Track unloading state separately in models store

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(webui): address PR review feedback for cancel model loading

- Remove duplicated cancel logic from ModelsSelector and ModelsSelectorSheet
  by deriving loading/cancelling state from the store (issue heiervang-technologies#1)
- Fix race condition: no longer set isLoadingModel=false before cancel
  completes, preventing brief UI flash (issue heiervang-technologies#2)
- Add MAX_CANCEL_POLL_ATTEMPTS (60) timeout to cancel polling loop
  to prevent infinite polling if server never transitions (issue heiervang-technologies#3)
- Replace div cancel buttons with proper <button> elements for
  keyboard accessibility and screen reader support (issue heiervang-technologies#4)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
feat(webui): rebrand to ht-llama.cpp and add turquoise/purple theme (h…
528ba29 
…eiervang-technologies#10)

- Rename all frontend references from "llama.cpp" to "ht-llama.cpp"
- Dark mode: turquoise-tinted backgrounds, purple-tinted text
- Light mode: inverted — turquoise backgrounds, purple text
- Add reverse spin animation utility class

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
feat: LoRA adapter auto-discovery and UI (heiervang-technologies#13, h…
5772904 
…eiervang-technologies#14)

Auto-discover LoRA adapters from the models directory by scanning GGUF
metadata (general.type = "adapter") and match them to models by
architecture. Adapters are loaded with --lora-init-without-apply so
they start disabled and can be toggled on via the UI.

Frontend adds a Popover-based LoRA dropdown in the chat action bar
(next to model selector) with multi-select checkboxes and scale inputs.
Includes "Show only matching" toggle to view all discovered adapters.
Works in both MODEL and ROUTER mode.

Backend changes:
- Add GGUF metadata scanning for adapter classification (preset.cpp)
- Auto-inject matching LoRA adapters into child process args (server-models.cpp)
- Include discovered adapters in /v1/models response
- Fix router proxy for /lora-adapters POST (array body fallback to query param)

Frontend changes:
- New LoraAdapters popover component with checkbox multi-select
- LoRA service with router mode support (query param routing)
- Reactive store with toggle, scale, change tracking, apply
- Integration in ChatFormActions bar and chat completion requests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
perf(cuda): add float4-vectorized RMS norm kernel
bd26c78 
Use 128-bit (float4) vectorized loads and stores in the RMS
normalization CUDA kernel to improve memory throughput on Ampere+
GPUs.  Each thread processes 4 floats per iteration, reducing the
total number of global memory transactions by ~4x and better
saturating memory bandwidth.

The vectorized path activates when ncols is divisible by 4 and
rows are contiguous (stride_row == ncols).  When these conditions
are not met the existing scalar kernel is used as a fallback, so
correctness is preserved for all tensor shapes.

The mathematical operation is unchanged -- output is bit-exact
compared to the scalar kernel.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
perf(cuda): reduce redundant per-token synchronizations
4df1fc5 
Remove unnecessary CUDA synchronization points that add latency without
providing correctness benefits:

1. mul_mat_id: remove cudaStreamSynchronize after H2D async copy of
   sorted IDs buffer - the subsequent get_rows_cuda kernel runs on the
   same stream, so CUDA stream ordering already guarantees completion.

2. split_buffer_set_tensor: replace N redundant cudaStreamSynchronize
   calls (looped once per device count) with a single sync - all async
   H2D copies use the same cudaStreamPerThread.

3. split_buffer_get_tensor: same fix as above for D2H copies.

These are scheduling-only changes with bit-exact output. Safe for
multi-GPU (split buffer sync still waits for all copies) and single-GPU.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
perf(cpu): add NUMA-aware 1:1 thread-to-core pinning with CCX locality
25da21f 
Add a new GGML_NUMA_STRATEGY_PIN that pins each compute thread to an
individual CPU core, ordered by L3 cache cluster (CCX/CCD) and NUMA node.
This maximizes memory locality for thread-cooperative operations like
matrix multiplication on multi-socket and multi-CCX systems (AMD EPYC,
Threadripper, Intel multi-socket).

Key changes:
- Detect L3 cache topology via /sys/devices/system/cpu/cpuN/cache/index3/
- Build optimal pin ordering: group cores by shared L3, then by NUMA node
- 1:1 thread pinning via pthread_setaffinity_np (single core per thread)
- Memory interleave policy via set_mempolicy syscall on multi-node systems
- Enable via GGML_NUMA_PIN=1 env var or --numa pin CLI flag
- Works on single-node multi-CCX systems (e.g. Threadripper)
- Graceful fallback: no libnuma dependency, compile-time Linux detection
- Scheduling-only change: bit-exact output preserved

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
perf(cpu): add software prefetching and cache-line alignment for quan…
6cf3d2a 
…tized matmul

Add explicit software prefetch hints to the CPU quantized matrix
multiplication hot loops to improve cache utilization and reduce
memory stalls during inference.

Changes:
- Add portable GGML_PREFETCH_T0/T1 macros: uses _mm_prefetch on x86
  (SSE/AVX/AVX2/AVX512) and __builtin_prefetch on ARM NEON and other
  GCC/Clang targets, with no-op fallback for unsupported compilers
- Prefetch src0 rows 4 iterations ahead in the inner vec_dot loop,
  covering 2-3 cache lines per row to hide memory latency
- Prefetch next src1 column data while processing current column
- Prefetch src0 data for the next block at block boundaries
- Align tmp accumulator arrays to cache-line boundaries (64 bytes)
  to prevent false sharing between threads
- Apply same optimizations to mul_mat_id (MoE expert dispatch)

All changes are hint-only (prefetch instructions are non-binding)
and produce bit-exact identical output. No algorithmic changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei @claude
perf(cpu): add software prefetching and cache-line alignment for quan…
a89183e 
…tized matmul

Add explicit software prefetch hints to the CPU quantized matrix
multiplication hot loops to improve cache utilization and reduce
memory stalls during inference.

Changes:
- Add portable GGML_PREFETCH_T0/T1 macros: uses _mm_prefetch on x86
  (SSE/AVX/AVX2/AVX512) and __builtin_prefetch on ARM NEON and other
  GCC/Clang targets, with no-op fallback for unsupported compilers
- Prefetch src0 rows 4 iterations ahead in the inner vec_dot loop,
  covering 2-3 cache lines per row to hide memory latency
- Prefetch next src1 column data while processing current column
- Prefetch src0 data for the next block at block boundaries
- Align tmp accumulator arrays to cache-line boundaries (64 bytes)
  to prevent false sharing between threads
- Apply same optimizations to mul_mat_id (MoE expert dispatch)

All changes are hint-only (prefetch instructions are non-binding)
and produce bit-exact identical output. No algorithmic changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei
Merge remote-tracking branch 'hai-pilgrim/perf/reduce-cuda-syncs' int…
245994f 
…o pilgrim/accumulated-optimizations
@marksverdhei
Merge remote-tracking branch 'hai-pilgrim/perf/numa-thread-pinning' i…
3514f16 
…nto pilgrim/accumulated-optimizations
@marksverdhei
Merge remote-tracking branch 'hai-pilgrim/perf/cpu-prefetch-blocking'…
d9f1a41 
… into pilgrim/accumulated-optimizations
@marksverdhei @claude
perf(vulkan): add shared-memory staging MatVec kernel
c29741d 
Implement a two-phase shared-memory staging approach for matrix-vector
multiplication in the Vulkan backend, decoupling A-matrix dequantization
from dot-product computation:

- Phase 1 (DECODE): All threads cooperatively load and dequantize a tile
  of the A-matrix into shared memory as f32 values
- Phase 2 (DOT): Each thread computes dot products from the staged shared
  memory tile against B-vector elements

This separation improves GPU utilization by allowing the memory-bound
dequantization and compute-bound dot products to be handled independently,
which is particularly beneficial on architectures with separate memory and
compute units (e.g., Intel Arc GPUs where upstream PR ggml-org#20897 showed >2.5x
token generation speedup).

The new kernel is selected automatically for single-column matvec
operations (token generation) when shared memory is sufficient. Falls
back to the existing fused kernel for unsupported types or insufficient
shared memory. Supports standard quant types (q4_0, q4_1, q5_0, q5_1,
q8_0), float types (f32, f16, bf16), and iq4/mxfp4 types.

Bit-exact output: same computation, different memory access pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei
Merge remote-tracking branch 'hai-pilgrim/perf/vulkan-shmem-matvec' i…
c8186a2 
…nto pilgrim/accumulated-optimizations
@marksverdhei @claude
perf(cuda): fused SiLU-multiply kernel for SwiGLU MLP blocks
b015e20 
Add a float4-vectorized fused kernel that computes SiLU(gate) * up in a
single pass, replacing the generic scalar unary_gated_op_kernel on the
hot path of Llama-style SwiGLU FFN blocks.

The vectorized kernel processes 4 elements per thread via float4
loads/stores, reducing memory transactions and eliminating function-
pointer indirection.  A scalar tail kernel handles the remaining 0-3
elements.  The fast path activates automatically for contiguous f32
tensors (the common case); f16 and strided layouts fall back to the
existing generic kernel.

Bit-exact: same mathematical operations (x / (1 + exp(-x)) * up),
just fused scheduling with wider memory access.

Based on Deep Kernel Fusion (arXiv 2602.11808) showing 9.7-13.2%
improvement from fusing element-wise ops in SwiGLU MLP blocks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei
Merge remote-tracking branch 'hai-pilgrim/perf/fused-silu-mul' into p…
9f1bfd4 
…ilgrim/accumulated-optimizations
@marksverdhei @claude
perf(cuda): double-buffered CUDA graph overlap for 3-7% throughput im…
ada00bc 
…provement

Implement CPU-side CUDA graph building overlap using double-buffering.
While the GPU executes the current graph instance, the CPU builds/updates
a secondary instance for the next call. On subsequent calls, instances are
swapped so the pre-built one launches immediately.

This eliminates the CPU-side cudaGraphExecUpdate blocking time from the
critical path, overlapping it with GPU execution instead.

- Opt-in via GGML_CUDA_GRAPH_OVERLAP=1 environment variable
- Bit-exact output: same graphs, only scheduling differs
- Properly invalidates overlap instance on warmup reset or graph recapture
- No impact when disabled (default)

Inspired by upstream PR ggml-org#11867 which showed 2.5-7% prompt eval and
2.8-3.6% token generation improvement.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marksverdhei
Merge remote-tracking branch 'hai-pilgrim/perf/cuda-graph-overlap' in…
99ab880 
…to pilgrim/accumulated-optimizations
@marksverdhei
fix(cpu): guard prefetch with minimum row size to avoid regression on…
e83c4bc 
… small models
@marksverdhei marksverdhei force-pushed the ht branch 2 times, most recently from 2189217 to 65d629d Compare April 5, 2026 13:17
@marksverdhei marksverdhei force-pushed the ht branch 2 times, most recently from 6846da3 to 139f68e Compare April 12, 2026 09:32
@marksverdhei

marksverdhei commented Apr 27, 2026

Copy link
Copy Markdown

Closing — head branch was deleted from remote. Reopen with a fresh branch if the work is still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hai-pilgrim @marksverdhei