ROCm/HIP compatibility: managed memory fallback, FP8 path, UMA memory calc#6
Closed
JDL440 wants to merge 169 commits into
Closed
ROCm/HIP compatibility: managed memory fallback, FP8 path, UMA memory calc#6JDL440 wants to merge 169 commits into
JDL440 wants to merge 169 commits into
Conversation
* qkv may not always be contiguous * cont : make the cont conditional --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : add option to save memory in device buffers * tests : extend llama-save-load-state
…-org#22538) Previously, unknown tool names passed via --tools were silently ignored. Now the server validates each tool name at startup and exits with an error if an unrecognized tool is specified, listing the available tools. Assisted-by: llama.cpp:local pi
* common : only load backends when required Signed-off-by: Adrien Gallouët <angt@huggingface.co> * llama : call ggml_backend_load_all() directly from llama_backend_init() Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Add ggml_backend_load_all() where llama_backend_init() is not used Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…gml-org#22702) common/arg.cpp:3719:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn] 3719 | [](common_params & /*params*/, int /*value*/) { | ^ common/arg.cpp:3726:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn] 3726 | [](common_params & /*params*/, int /*value*/) { | ^ common/arg.cpp:3733:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn] 3733 | [](common_params & /*params*/, int /*value*/) { | ^ common/arg.cpp:3740:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn] 3740 | [](common_params & /*params*/, int /*value*/) { | ^ common/arg.cpp:3747:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn] 3747 | [](common_params & /*params*/, int /*value*/) { | ^ Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Store the last graph uid and compare against it to determine if the same graph is being computed.
* opencl: refactor adreno q4_0 gemm/gemv dispatch * opencl: refactor q4_0 gemm/gemv loading, use consistent names * opencl: use consistent name for adreno q8_0 gemm/gemv * opencl: use consistent names for adreno q4_0 gemm/gemv * opencl: simplify adreno q4_0 set_tensor * opencl: refactor q4_0 get_tensor
* hex-mm: process m-tail rows on HMX instead of HVX * hmx-mm: unroll and optimize padded activation loop --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
…--fit (ggml-org#22688) * ggml : report estimated OpenCL memory for --fit Signed-off-by: Florian Reinle <f.reinle@otec.de> * ggml : estimated OpenCL memory backend integrated Signed-off-by: Florian Reinle <f.reinle@otec.de> --------- Signed-off-by: Florian Reinle <f.reinle@otec.de>
…2597) * add filter_tensors classmethod * remove language_model * fix parts validation
…gml-org#22719) * refactor: Remove Google favicon utility * fix: MCP Server favicon * refactor: Cleanup * refactor: MCP Server Information * fix: Fix MCP Settings UI * refactor: Cleanup
* convert : ignore non-language tensors for Gemma4Model
This commit adds a check to make sure only text language tensors are
handled in filter_tensors.
The motivation is that currently when trying to convert a Gemma4 model
the following error occurs:
```console
(venv) $ ./convert-gemma.sh
INFO:hf-to-gguf:Loading model: gemma-4-E2B-it
INFO:hf-to-gguf:Model architecture: Gemma4ForConditionalGeneration
INFO:hf-to-gguf:gguf: indexing model part 'model.safetensors'
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:rope_freqs.weight, torch.float32 --> F32, shape = {256}
Traceback (most recent call last):
File "/home/danbev/work/llama.cpp/./convert_hf_to_gguf.py", line 13752, in <module>
main()
File "/home/danbev/work/llama.cpp/./convert_hf_to_gguf.py", line 13746, in main
model_instance.write()
File "/home/danbev/work/llama.cpp/./convert_hf_to_gguf.py", line 945, in write
self.prepare_tensors()
File "/home/danbev/work/llama.cpp/./convert_hf_to_gguf.py", line 805, in prepare_tensors
for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
File "/home/danbev/work/llama.cpp/./convert_hf_to_gguf.py", line 7925, in modify_tensors
yield from super().modify_tensors(data_torch, name, bid)
File "/home/danbev/work/llama.cpp/./convert_hf_to_gguf.py", line 7290, in modify_tensors
yield from super().modify_tensors(data_torch, name, bid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/danbev/work/llama.cpp/./convert_hf_to_gguf.py", line 579, in modify_tensors
new_name = self.map_tensor_name(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/danbev/work/llama.cpp/./convert_hf_to_gguf.py", line 572, in map_tensor_name
raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'model.embed_vision.embedding_projection.weight'
```
* add forgotten embed_vision and embed_audio
* improve
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* feat: migrate to PEP 621 and add uv support * fix: remove upper bound on protobuf * remove poetry.lock and uv.lock * fix/add torch dependency version and markers * fix dev-dependency deprecation warning * gguf-py : update python version requirement to 3.10 --------- Co-authored-by: David Huggins-Daines <dhd@dhd.ecolingui.ca> Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
…gml-org#22101) * mtmd: add granite-speech support (ibm-granite/granite-4.0-1b-speech) Conformer encoder with Shaw relative position encoding, QFormer projector, log-mel spectrogram with frame stacking. Encoder uses GLU gating, folded batch norm, and SSM depthwise conv. QFormer compresses encoder output via windowed cross-attention (window=15, queries=3) into the LLM embedding space. Audio preprocessing: reflect-padded STFT, 80-bin mel filterbank, dynamic range compression, 2x frame stacking (80->160 mel). GGUF converter handles batch norm folding at export time, fused K/V split, and Conv1d weight reshaping. Tested against HF transformers reference: token-for-token match on 30s/60s audio clips with greedy decoding. * mtmd: rename gs_ prefixed tensors to generic/architecture names * mtmd: use tensor_mapping.py for all granite_speech tensors * convert: fold GraniteSpeechTextModel into GraniteModel * mtmd: replace n_layer hack with explicit has_standard_layers flag * mtmd: replace hardcoded magic numbers with GGUF hparams for granite speech * mtmd: align KEY_A_ define spacing * convert: register GraniteModel for GraniteSpeechForConditionalGeneration * convert: fix ty type-check for GraniteSpeechMmprojModel registration * mtmd: align TN_ define spacing * mtmd: use generic layer loop for granite speech tensor loading * mtmd: merge qformer_proj_layer into clip_layer * mtmd: granite_speech remove redundant ggml_build_forward_expand on inputs * mtmd: granite_speech add comment explaining why build_attn is not used * mtmd: granite_speech hard-code eps in cpp, remove from GGUF metadata * gguf: add spacing between granite_speech tensor mapping blocks * mtmd: make generic audio layer_norm_eps read optional * mtmd: granite_speech keep encoder eps in GGUF, only hard-code projector eps * mtmd: align defines and struct fields in clip-impl.h and clip-model.h * mtmd: fix alignment and ordering issues across granite speech files * convert: granite_speech use filter_tensors instead of modify_tensors for skipping
* gguf-py : bump version to 0.19.0 * bump poetry --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* common: do not fit to unknown device memory Signed-off-by: Florian Reinle <f.reinle@otec.de> * common: preserve host fallback for non-GPU fit devices Signed-off-by: Florian Reinle <f.reinle@otec.de> * common: keep unknown GPU fit memory at zero Signed-off-by: Florian Reinle <f.reinle@otec.de> --------- Signed-off-by: Florian Reinle <f.reinle@otec.de>
* model: don't crash on unsupported architecture * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Support MiniCPM-V 4.6 in new branch Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix code bug Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix pre-commit Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix convert Signed-off-by: tc-mb <tianchi_cai@icloud.com> * rename clip_graph_minicpmv4_6 Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use new TYPE_MINICPMV4_6 Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use build_attn to allow flash attention support Signed-off-by: tc-mb <tianchi_cai@icloud.com> * no use legacy code, restored here. Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use the existing tensors name Signed-off-by: tc-mb <tianchi_cai@icloud.com> * unused ctx->model.hparams.minicpmv_version Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use n_merge for slice alignment Signed-off-by: tc-mb <tianchi_cai@icloud.com> * borrow wa_layer_indexes for vit_merger insertion point Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix code style Signed-off-by: tc-mb <tianchi_cai@icloud.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * use filter_tensors and add model.vision_tower Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix chkhsh Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix type check Signed-off-by: tc-mb <tianchi_cai@icloud.com> --------- Signed-off-by: tc-mb <tianchi_cai@icloud.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
The error:
./examples/sycl/test.sh: line 122: level_zero:${$GGML_SYCL_DEVICE}: bad
substitution
was thrown whenever the user used this command:
./examples/sycl/test.sh -mg 0
Fix is to get rid of a dollar sign.
…gml-org#22773) * add fill-mode-forwards * generated diffs
* codeowners : add ZenDNN backend codeowner * codeowners : fix zendnn owners to use individual github handles
* webui: fix ?model= URL param race in router mode * chore: update webui build output
* add mimo-v2.5 support * mimo-v2.5: fix modify_tensors row split * mimi-v2.5: forgot `add_attn_value_scale` plumbing * mimi-v2.5: fix tp dequant to detect tp rows * mimo-v2.5: fix TP iteration to be descending * mimo-v2.5: fix comment * mimo-v2.5: retain fused qkv * mimo-v2.5: missed the attn_value scale during merge * mimo-v2.5: fused QKV needs contiguous for scaling attention value * mimo-v2.5: move `speech_embeddings.` to TextModel filter_tensors * Update src/llama-hparams.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/mimo2.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/mimo2.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/mimo2.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * mimo-v2.5: include MTP weights in gguf --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Adds Metal implementations + dispatch for the five DeepSeek-V4-Flash ggml ops: DSV4_HC_SPLIT_SINKHORN, DSV4_HC_WEIGHTED_SUM, DSV4_HC_EXPAND, DSV4_FP8_KV_QUANTIZE, DSV4_ROPE_TAIL. Additive only, bounded to ggml/src/ggml-metal/* (7 files, +913/-0). No DeepSeek Sparse Attention / lightning-indexer / flash-attn-top-k Metal code. Validated on Apple M3 Ultra (Metal, reference platform): coherent V4 inference with DSA entirely absent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CUDA implementations + dispatch for the five DeepSeek-V4-Flash ggml ops (DSV4_HC_SPLIT_SINKHORN, DSV4_HC_WEIGHTED_SUM, DSV4_HC_EXPAND, DSV4_FP8_KV_QUANTIZE, DSV4_ROPE_TAIL), plus the V4-coupled multi-GPU correctness fixes in ggml-cuda.cu (split-buffer DSV4 exception, view_src cross-device dispatch refusal, env-gated peer-copy debug) and the software-FP16 dequantize_V<half> fallback in fattn-common.cuh. Additive, bounded to ggml/src/ggml-cuda/* (12 files, +1142/-2; the 2 deletions are the intentional split-buffer guard replacement). No DeepSeek Sparse Attention / lightning-indexer / flash-attn-top-k code. The dequantize_V<half> fallback is code byte-identical to the merged PR-series fix d4e21f0; one rationale comment was generalized to not name a kernel that does not exist on this DSA-free base. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the DSA-free V4 source of truth up to date with ggml-org master (18d1717 -> 0253fb2). 22/23 overlapping core files auto-merged. Sole content conflict: src/llama-arch.cpp LLM_TENSOR_INFOS. Resolved by keeping the V4 hyperconnection / compressed-KV tensor->op mappings (INDEXER_COMPRESSOR_*, HC_*, FFN_GATE_TID2EID, *_APE) and adopting upstream's NextN/MTP reclassification (LAYER_OUTPUT -> LAYER_REPEATING, the loader-block-index fault fix). V4 functionally ignores NextN/MTP (reserved for future MTP), so upstream's more-robust shared-infra classification is the correct up-to-date choice. DSA-exclusion intact: lightning-indexer / flash_attn-topk / kv_cache_dsa absent; the 5 V4 layer commits unchanged. Revalidation on gpudual (CUDA build + DSV4 19/19 + coherent decode) before push. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Silent semantic conflict from the +20 upstream merge: origin/master added a new `uint32_t n_rs_seq` parameter (position 13) to the llama_memory_hybrid_iswa constructor. Git auto-merged the header and the V4 call site separately with no text conflict, leaving the V4 create_memory() path passing 16 args to a 17-arg ctor (filter_attn landed on `bool unified`, a std::function->bool error). Pass `cparams.n_rs_seq` at position 13, matching the convention of the two non-V4 hybrid_iswa/hybrid call sites in this file. C++ side (llama, llama-completion, test-backend-ops) builds clean locally; gpudual CUDA revalidation follows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…memory calc - ggml-cuda.cu: add fallback from cudaMallocManaged to cudaMalloc when managed allocation fails (OOM, host registration limits on APUs). Also fix UMA free-memory calculation to include SwapFree for paged-out weights on high-swap systems. - dsv4-fp8-kv-quantize.cu/cuh: switch from __CUDA_ARCH__ gate to FP8_AVAILABLE macro so HIP targets get native FP8 paths. Fix __shfl_xor_sync call with explicit WARP_SIZE parameter. - deepseek4.cpp: clamp n_comp_visible to cache capacity in graph_reserve to prevent ubatch over-count during context init.
github-actions
Bot
added
documentation
Author
|
Closing this polluted PR — will open a clean one with only the relevant changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ROCm/HIP Compatibility Fixes
Fixes GPU offloading on AMD APUs (e.g. Radeon 8060S / Strix Halo) where managed/UMA memory allocation fails due to host-page registration limits.
Changes
ggml-cuda.cu: Add fallback from
cudaMallocManagedtocudaMallocwhen managed allocation fails (OOM, RLIMIT_MEMLOCK, unsupported platform). Fix UMA free-memory calculation to include SwapFree for paged-out weights on high-swap systems.dsv4-fp8-kv-quantize.cu/.cuh: Switch from
__CUDA_ARCH__ >= 890gate toFP8_AVAILABLEmacro so HIP targets get native FP8 paths. Fix__shfl_xor_synccall with explicit WARP_SIZE parameter.deepseek4.cpp: Clamp
n_comp_visibleto cache capacity in graph_reserve to prevent ubatch over-count during context init.Build (ROCm/HIP on gfx1151)
Uses ROCm clang toolchain with libc++. At runtime:
export LD_LIBRARY_PATH=/path/to/libcxx/lib64:/path/to/build/binTesting
Verified on AMD Radeon 8060S (gfx1151, Ryzen AI MAX+ 395 APU, 128 GiB DDR5 with 32/96 system/VRAM split):
--no-mmap --ctx-size 16384Key findings:
cudaMallocManagedfails without of memoryon this APU due to RLIMIT_MEMLOCK. The fallback tocudaMallocresolves it.--no-mmapis critical for models >32 GiB: mmap funnels through the system RAM page cache (32 GiB), thrashing. Without mmap, data reads directly into VRAM (96 GiB).--ctx-size 16384needed for DSv4: full 1M context KV cache (~27 GiB) + model (81 GiB) = 108 GiB > 96 GiB VRAM. 16K is a safe default.Fallback log evidence:
Related Config
Launcher scripts, OpenCode config, and full architecture notes saved at: https://github.com/JDL440/BorrowTokens/tree/main/contrib/local-models