Misc. bug: CUDA memory leak with MTMD requests loop with Gemma3

[Cyrille37]

Hi.

Thanks a lot for sharing this so great ggml work 💌.

Name and Version

./build/bin/llama-cli --version

ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
version: 7971 (5fa1c19)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux NS5x-NS7xAU.localdomain 6.17.0-14-generic #14~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 15 15:52:10 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Which llama.cpp modules do you know to be affected?

llama-server

$ git log -1
commit 5fa1c190d9fc86c02698b730a2cb933195e19d96 (HEAD -> master, origin/master, origin/HEAD)
Author: Adrien Gallouët <angt@huggingface.co>
Date:   Sun Feb 8 09:06:45 2026 +0100
    rpc : update from common.cpp (#19400)    
    Signed-off-by: Adrien Gallouët <angt@huggingface.co>

$ export CUDA_VERSION=12.9 && cmake -B build -DGGML_CUDA=ON \
 -DCMAKE_CUDA_ARCHITECTURES="86;120" \
 -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-${CUDA_VERSION}/bin/nvcc \
 -DCMAKE_INSTALL_RPATH="/usr/local/cuda-${CUDA_VERSION}/lib64;\$ORIGIN"

$ cmake --build build --config Release -j 10

Command line

./llama.cpp/build/bin/llama-server --port 8012 \
 -m ~/Data/AI_ModelsVision/gemma-3-4b-it-Q4_K_M.gguf \
 --mmproj ~/Data/AI_ModelsVision/mmproj-model-f16.gguf \
 -c 0 --parallel 1

--parallel 1 does not change the count of requests before fail.

Problem description & steps to reproduce

The use case is object detection (MTMD) in small jpeg images (1.7Ko to 19ko) and llama-server fails with CUDA out of memory after ~700 requests.

I do iterations on many different images (thousands) so it should not be image that throws the fail but is it all the time around 700 / 780 requests.

Perhaps it should not be a RTX/CUDA/Driver error because I can relaunch the script and it works fine for another ~700 requests and no message in the system journal (unlike for rtx/cuda/driver errors).

My bug report is poor of precision, but ask me for operations, I'll try to apply them for report completion.

First Bad Commit

No response

Relevant log output

Logs

ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
build: 7971 (5fa1c190d) with GNU 13.3.0 for Linux x86_64
system info: n_threads = 4, n_threads_batch = 4, total_threads = 16

system_info: n_threads = 4 (n_threads_batch = 4) / 16 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

Running without SSL
init: using 15 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '~/Data/AI_ModelsVision/gemma-3-4b-it-Q4_K_M.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on

...
... running well for 700 request ...
...

./llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
CUDA error: out of memory
  current device: 0, in function ggml_cuda_graph_evaluate_and_capture at ./llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:3910
  cudaGraphInstantiate(&graph->instance, graph->graph, __null, __null, 0)
[New LWP 7622]
[New LWP 7621]
[New LWP 7620]
[New LWP 7619]
[New LWP 7618]
[New LWP 7617]
[New LWP 7616]
[New LWP 7615]
[New LWP 7614]
[New LWP 7613]
[New LWP 7612]
[New LWP 7611]
[New LWP 7610]
[New LWP 7609]
[New LWP 7608]
[New LWP 7607]
[New LWP 7606]
[New LWP 7605]
[New LWP 7604]
[New LWP 7603]
[New LWP 7598]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x000071e034310813 in __GI___wait4 (pid=9109, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: Aucun fichier ou dossier de ce nom
#0  0x000071e034310813 in __GI___wait4 (pid=9109, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30	in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x000071e03496e703 in ggml_print_backtrace () from ./llama.cpp/build/bin/libggml-base.so.0
#2  0x000071e03496e8ab in ggml_abort () from ./llama.cpp/build/bin/libggml-base.so.0
#3  0x000071e028963607 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from ./llama.cpp/build/bin/libggml-cuda.so.0
#4  0x000071e02897bbbb in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from ./llama.cpp/build/bin/libggml-cuda.so.0
#5  0x000071e02897ca7e in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from ./llama.cpp/build/bin/libggml-cuda.so.0
#6  0x000071e03498ae37 in ggml_backend_sched_graph_compute_async () from ./llama.cpp/build/bin/libggml-base.so.0
#7  0x000071e034abde01 in llama_context::graph_compute(ggml_cgraph*, bool) () from ./llama.cpp/build/bin/libllama.so.0
#8  0x000071e034abf8c4 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from ./llama.cpp/build/bin/libllama.so.0
#9  0x000071e034ac6daa in llama_context::decode(llama_batch const&) () from ./llama.cpp/build/bin/libllama.so.0
#10 0x000071e034ac881f in llama_decode () from ./llama.cpp/build/bin/libllama.so.0
#11 0x00005db023ec4a48 in server_context_impl::update_slots() ()
#12 0x00005db023f0c7ae in server_queue::start_loop(long) ()
#13 0x00005db023e25770 in main ()
[Inferior 1 (process 7597) detached]
Abandon (core dumped)

[LucaAmigoni]

I have a similar issue on Windows 11 using precompiled bins (llama-b8069-bin-win-cuda-13.1-x64), building from source seems to resolve. Note that git pull is ahead b8069, but I don't see resolutions for this in recent commits.
BTW, it seems that cache access (prompt or kv) triggers a new CUDA alloc of variable size depending on model, from 30 to 100MB.

git log -1
commit d612901 (HEAD -> master, origin/master, origin/HEAD)
Author: AesSedai 7980540+AesSedai@users.noreply.github.com
Date: Mon Feb 16 08:44:44 2026 -0800

perplexity: add proper batching (#19661)

Build with:

-G "Visual Studio 17 2022" ^
-DLLAMA_CURL=OFF ^
-DLLAMA_SERVER_SSL=OFF ^
-DGGML_NATIVE=ON ^
-DGGML_RPC=ON ^
-DLLAMA_BUILD_SERVER=ON ^
-DGGML_BACKEND_DL=OFF ^
-DGGML_AVX=ON ^
-DGGML_AVX2=ON ^
-DGGML_AVX512=ON ^
-DGGML_AVX512_VBMI=ON ^
-DGGML_AVX512_VNNI=ON ^
-DGGML_AVX512_BF16=OFF ^
-DGGML_MATMUL_INT8=ON ^
-DGGML_CUDA=ON ^
-DGGML_CUDA_GRAPH=ON ^
-DGGML_CUDA_USE_CUBLASLT=ON ^
-DGGML_CUDA_FORCE_CUBLAS=ON ^
-DGGML_LTO=ON ^
-DGGML_OPENMP=ON ^
-DGGML_FLASH_ATTN=ON ^
-DGGML_CUDA_FA_ALL_QUANTS=true

[osma]

I have the same issue as OP. I'm using Gemma3-4b-it model (Unsloth UD Q5_K_XL quantization) with mmproj-BF16.gguf and I'm analyzing images one at a time via API with a simple prompt. VRAM usage starts at ~4.7GB but slowly grows as llama-server processes API requests and eventually hits CUDA OOM at 6GB.
I have a RTX 3060 Laptop GPU. Using llama.cpp version 8057 on Linux Mint.

I think it has to do with processing images because I successfully processed 30k+ text prompts with the same Gemma model without loading mmproj.

[osma]

I tried to narrow this down and come up with a minimal example.

First, I recompiled latest llama.cpp with CUDA.

$ build/bin/llama-server --version
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes
version: 8100 (c0d043034)
built with GNU 13.3.0 for Linux x86_64

Then I started up llama-server with Unsloth Gemma3-4b-it and mmproj directly from HuggingFace using -hf option. With -c 50000 initial VRAM usage according to nvidia-smi is 5479MiB / 6144MiB .

build/bin/llama-server -hf unsloth/gemma-3-4b-it-GGUF -c 50000

Then I ran this test script that generates simple images with random text and sends them to the API endpoint with a prompt to OCR the text.

Test script

Requires pillow and openai libraries.

#!/usr/bin/env python

import base64
import io
import sys
from PIL import Image, ImageDraw
from openai import OpenAI
import random

# --------------------------------------------------------------------------- #
# Helper: Generate a random string
# --------------------------------------------------------------------------- #
def generate_random_string(length=10):
    """Generates a random string of alphanumeric characters."""
    characters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
    return ''.join(random.choice(characters) for _ in range(length))

# --------------------------------------------------------------------------- #
# Core: Generate and save the image
# --------------------------------------------------------------------------- #
def generate_image(text=None):
    """Generates a white background image with black text."""
    img = Image.new("RGB", (768, 512), "white")
    d = ImageDraw.Draw(img)

    # Add the text
    if text:
        x = 768 / 2
        y = 512 / 2
        d.text((x, y), text, fill="black", anchor="ms")

    return img

# --------------------------------------------------------------------------- #
# Core: Ask the VLM to classify the image
# --------------------------------------------------------------------------- #
def get_image_description(img):
    """Sends the image to the local vision model and returns the description."""
    client = OpenAI(
        base_url="http://localhost:8080/v1",
        api_key="foobar",
    )

    # Encode image to base64 in memory
    buffered = io.BytesIO()
    img.save(buffered, format="PNG")
    base64_image = base64.b64encode(buffered.getvalue()).decode("utf-8")

    # Prompt that includes the full descriptions of the palettes
    prompt = (
        "Read the text in this image. Return only the text."
    )

    response = client.chat.completions.create(
        model="gemma-3-4b-it",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{base64_image}",
                        },
                    },
                    {"type": "text", "text": prompt},
                ],
            }
        ],
    )

    # The model should return just the description
    return response.choices[0].message.content.strip()

# --------------------------------------------------------------------------- #
# CLI entry point
# --------------------------------------------------------------------------- #
def main():
    rounds = 1
    
    if len(sys.argv) > 1:
        rounds = int(sys.argv[1])

    for round in range(rounds):
        try:
            text = generate_random_string()

            # Generate the image in memory
            img = generate_image(text)

            # Get the image description
            description = get_image_description(img)

            print(f"Round {round+1}: {description}")

        except Exception as e:
            print(f"Error processing image: {e}", file=sys.stderr)

if __name__ == "__main__":
    main()

Here is the output when tried to run it for 100 rounds:

$ ./desc-dummy-img.py 100
Round 1: hbagZqj3TG
Round 2: xrx9vNuqKF
Round 3: wCtIdqbxPrV
Round 4: tuekEgH8VW
Round 5: DaWJdgs3aR
Round 6: IqWWxqqHtfY
Round 7: Bp43PANIBT
Round 8: gN2wgejs8u
Round 9: b3lvbtbrW8
Round 10: slIMc36tr
Round 11: KkFUb3Xpbg
Round 12: OuHJinWk28
Round 13: gyrmZ9 TFN8
Round 14: D36Z S7M94h
Round 15: d88c2JbDYG
Round 16: IYO Sg40kJY
Error processing image: Connection error.
Error processing image: Connection error.
Error processing image: Connection error.
[...]

So llama.cpp is able to process 16 requests before crashing due to CUDA OOM when VRAM usage hits 6GB, which is what I have on my RTX 3060 Laptop GPU.

Final output from llama.cpp

slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = 497599004465
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 155 | processing task, is_child = 0
slot update_slots: id  3 | task 155 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 280
slot update_slots: id  3 | task 155 | n_tokens = 5, memory_seq_rm [5, end)
srv  process_chun: processing image...
encoding image slice...
image slice encoded in 944 ms
decoding image batch 1/1, n_tokens_batch = 256
sched_reserve: reserving ...
sched_reserve:      CUDA0 compute buffer size =   517.12 MiB
sched_reserve:  CUDA_Host compute buffer size =   117.02 MiB
sched_reserve: graph nodes  = 1369
sched_reserve: graph splits = 2
sched_reserve: reserve took 93.95 ms, sched copies = 1
/home/ozone/git/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
CUDA error: out of memory
  current device: 0, in function ggml_cuda_graph_evaluate_and_capture at /home/ozone/git/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:3891
  cudaGraphInstantiate(&graph->instance, graph->graph, __null, __null, 0)
[New LWP 1610011]
[New LWP 1610010]
[New LWP 1610009]
[New LWP 1610008]
[New LWP 1610007]
[New LWP 1610006]
[New LWP 1610005]
[New LWP 1610004]
[New LWP 1610003]
[New LWP 1610002]
[New LWP 1610001]
[New LWP 1610000]
[New LWP 1609999]
[New LWP 1609998]
[New LWP 1609997]
[New LWP 1609996]
[New LWP 1609995]
[New LWP 1609994]
[New LWP 1609985]
[New LWP 1609980]
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007c8b5ed10813 in __GI___wait4 (pid=1610805, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: Tiedostoa tai hakemistoa ei ole
#0  0x00007c8b5ed10813 in __GI___wait4 (pid=1610805, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30	in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x00007c8b5f3b1703 in ggml_print_backtrace () from /home/ozone/git/llama.cpp/build/bin/libggml-base.so.0
#2  0x00007c8b5f3b18ab in ggml_abort () from /home/ozone/git/llama.cpp/build/bin/libggml-base.so.0
#3  0x00007c8b57165ad7 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/ozone/git/llama.cpp/build/bin/libggml-cuda.so.0
#4  0x00007c8b5717b7dd in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from /home/ozone/git/llama.cpp/build/bin/libggml-cuda.so.0
#5  0x00007c8b5717c0ee in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/ozone/git/llama.cpp/build/bin/libggml-cuda.so.0
#6  0x00007c8b5f3cde47 in ggml_backend_sched_graph_compute_async () from /home/ozone/git/llama.cpp/build/bin/libggml-base.so.0
#7  0x00007c8b5f502741 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/ozone/git/llama.cpp/build/bin/libllama.so.0
#8  0x00007c8b5f504244 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/ozone/git/llama.cpp/build/bin/libllama.so.0
#9  0x00007c8b5f50b816 in llama_context::decode(llama_batch const&) () from /home/ozone/git/llama.cpp/build/bin/libllama.so.0
#10 0x00007c8b5f50d2af in llama_decode () from /home/ozone/git/llama.cpp/build/bin/libllama.so.0
#11 0x00007c8b5f7b6cf8 in mtmd_helper_decode_image_chunk () from /home/ozone/git/llama.cpp/build/bin/libmtmd.so.0
#12 0x00007c8b5f7b8280 in mtmd_helper_eval_chunk_single () from /home/ozone/git/llama.cpp/build/bin/libmtmd.so.0
#13 0x00005e2feeed746b in server_tokens::process_chunk(llama_context*, mtmd_context*, unsigned long, int, int, unsigned long&) const ()
#14 0x00005e2feef3269c in server_context_impl::update_slots() ()
#15 0x00005e2feef79bde in server_queue::start_loop(long) ()
#16 0x00005e2feee92bf0 in main ()
[Inferior 1 (process 1609979) detached]

If I start with a smaller context (-c 30000) then initial VRAM usage is 5089MiB / 6144MiB and I can do 41 requests before hitting OOM. Calculating from this, I think that llama.cpp leaks around 25MB VRAM on every request.

What I have tried as well:

• Testing with the same image/prompt over and over. llama.cpp will just return a cached response. There is no memory leak. Therefore the test script has to generate a different image for every request.
• Text-only prompts without an image will not cause VRAM usage to grow.
• Loading the model without mmproj will cause text+image requests to fail with 500 errors.

What I have not tried:

• Other VLMs than Gemma3-4b-it

[Cyrille37]

Nice report, thanks 👌

[osma]

Some more observations:

1. I tried SmolVLM-2B with the command build/bin/llama-server -hf ggml-org/SmolVLM2-2.2B-Instruct-GGUF. VRAM usage is stable at 3445 MiB even after 100 requests with the test script. Result quality is pretty bad: some of the responses are long strings of numbers or gibberish. But that's a model quality issue, not a VRAM issue.

2. I also tried Qwen3-VL-4B-Instruct with a 10000 token context using the command build/bin/llama-server -hf unsloth/Qwen3-VL-4B-Instruct-GGUF -c 10000. VRAM usage is stable at 5425MiB after 100 requests. It's also amazingly fast - around six times faster than the similarly sized Gemma3 4B!

3. Finally, I tested Gemma3 again with a smaller test image (256x128 instead of 768x512). The result did not change; with a 30000 context I can still make 41 requests before OOM crashing, so the VRAM leak is apparently not affected by image size.

Conclusion: This VRAM leaking does not apply to all VLMs equally; it might be specific to Gemma.

@Cyrille37 Perhaps you could update the issue title to reflect these findings? E.g. "CUDA memory leak on Gemma3 processing images" or something along those lines?

[ORippler]

Does #19754 or alternatively disabling cuda graphs via GGML_CUDA_DISABLE_GRAPHS fix this?

[osma]

@ORippler Thank you for the suggestions.

I tried again with the test case in #19639 (comment) (anyone can try this BTW). I'm using llama.cpp version: 8179 (ecbcb7e) which already contains #19754.

1. First I confirmed that the bug is still present (although now I can generate 40 images before crashing, not just 16)
2. Setting GGML_CUDA_DISABLE_GRAPHS=1 prevents the issue: VRAM usage is stable at 5531MiB according to nvidia-smi

[jstrot]

@ORippler Thanks!

Indeed GGML_CUDA_DISABLE_GRAPHS=1 prevents the issue for me as well.

---

llama.cpp: Docker ghcr.io/ggml-org/llama.cpp:server-cuda12-b8667 (today's latest)
Model: unsloth/gemma-4-E4B-it-GGUF UD-Q8_K_XL + mmproj-BF16
GPU: RTX 5060 Ti
Task: image analysis