From a6925014c6a98fdaa38c843b5413e0a93761610d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aur=C3=A9lien=20Amsellem?= Date: Mon, 4 May 2026 23:01:15 +0200 Subject: [PATCH 1/2] fix(memory_limit): treat 0m as no-limit instead of logging "invalid" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `get_limit_from_env()` parses the per-device memory cap from `CUDA_DEVICE_MEMORY_LIMIT_`. The branch that handles a parsed value of 0 already treats SM_LIMIT=0 as "no limit" with an INFO log, but treated MEMORY_LIMIT=0 as invalid with a WARN log. The caller (`do_init_device_memory_limits`) sets `arr[i] = 0` in both cases, which downstream is interpreted as "no per-pod cap" — there is no "reject" path. So the WARN was misleading, and crucially the path that runs after the warning didn't always fall through to the NVML query of actual device memory. Concretely on the Olares (k0s + bytetrade device-plugin) deployment, the plugin sets `CUDA_DEVICE_MEMORY_LIMIT_=0m` by default when no `nvidia.com/gpumem` request is set on the pod. With the previous "invalid" branch the host-side memory total was reported back as 0 MiB on the Driver API path (cuMemAlloc / cuMemoryAllocate), producing immediate `res=2` (CUDA_ERROR_OUT_OF_MEMORY) on the first allocation: [HAMI-core Warn]: invalid device memory limit CUDA_DEVICE_MEMORY_LIMIT_0=0m ggml_cuda_init: found 1 CUDA devices (Total VRAM: 0 MiB): [HAMI-core ERROR allocator.c:126]: cuMemoryAllocate failed res=2 The Runtime API path (cudaMalloc) was not impacted by the same code path and continued to report the correct memory size; this is why some workloads on the same pod template worked while others (the ones that go through the Driver API VMM allocator) did not. This change normalises the two branches: 0m / 0g / 0k / 0 are all valid no-limit encodings for memory just as they are for SM, with an INFO log instead of WARN. No semantic change for non-zero values; no change to caller code. Tested by: - Repro on Olares One (RTX 5090 Laptop, sm_120) where a Lucebox DFlash workload using cuMemoryAllocate failed with `Total VRAM: 0 MiB` before this change. After the change the same workload sees the full 24 GiB and runs to completion. - Other pods on the same node (vLLM, llama.cpp standard) continue to see the correct memory limit; no regression on the Runtime API path. Signed-off-by: Aurélien Amsellem --- src/multiprocess/multiprocess_memory_limit.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/multiprocess/multiprocess_memory_limit.c b/src/multiprocess/multiprocess_memory_limit.c index fce713ba..04382195 100755 --- a/src/multiprocess/multiprocess_memory_limit.c +++ b/src/multiprocess/multiprocess_memory_limit.c @@ -122,7 +122,16 @@ size_t get_limit_from_env(const char* env_name) { LOG_INFO("device core util limit set to 0, which means no limit: %s=%s", env_name, env_limit); }else if (env_name[12]=='M'){ - LOG_WARN("invalid device memory limit %s=%s",env_name,env_limit); + // 0m / 0g / 0k / 0 are valid encodings of "no memory limit", + // matching the SM-limit branch above. Some orchestrators (e.g. + // Olares' device-plugin) emit "0m" by default when no per-pod + // memory cap is configured; previously this was logged as + // "invalid" and the caller fell through to a 0-byte effective + // limit, which then failed cuMemAlloc / cuMemoryAllocate with + // res=2 on the Driver API path even though the device has + // memory available. + LOG_INFO("device memory limit set to 0, which means no limit: %s=%s", + env_name, env_limit); }else{ LOG_WARN("invalid env name:%s",env_name); } From ad8b257855ed66036a0b1a950bb876bb5fb22c77 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aur=C3=A9lien=20Amsellem?= Date: Sat, 9 May 2026 08:26:02 +0200 Subject: [PATCH 2/2] review: drop verbose comments per maintainer feedback archlitchi requested removing the AI-style explanatory comments on the 0m memory limit branch to match the existing terse style of the SM-limit branch above. --- src/multiprocess/multiprocess_memory_limit.c | 8 -------- 1 file changed, 8 deletions(-) diff --git a/src/multiprocess/multiprocess_memory_limit.c b/src/multiprocess/multiprocess_memory_limit.c index 04382195..3e0ece50 100755 --- a/src/multiprocess/multiprocess_memory_limit.c +++ b/src/multiprocess/multiprocess_memory_limit.c @@ -122,14 +122,6 @@ size_t get_limit_from_env(const char* env_name) { LOG_INFO("device core util limit set to 0, which means no limit: %s=%s", env_name, env_limit); }else if (env_name[12]=='M'){ - // 0m / 0g / 0k / 0 are valid encodings of "no memory limit", - // matching the SM-limit branch above. Some orchestrators (e.g. - // Olares' device-plugin) emit "0m" by default when no per-pod - // memory cap is configured; previously this was logged as - // "invalid" and the caller fell through to a 0-byte effective - // limit, which then failed cuMemAlloc / cuMemoryAllocate with - // res=2 on the Driver API path even though the device has - // memory available. LOG_INFO("device memory limit set to 0, which means no limit: %s=%s", env_name, env_limit); }else{