feat(vulkan): Vulkan implicit layer to enforce per-pod GPU memory budget on vkAllocateMemory

Makefile

@@ -13,7 +13,7 @@ build-in-docker:
 		-e DEBIAN_FRONTEND=noninteractive \
 		nvidia/cuda:12.9.1-cudnn-devel-ubuntu20.04 \
 		sh -c "apt-get -y update && \
-           apt-get -y install cmake git && \
+           apt-get -y install cmake git libvulkan-dev && \
            git config --global --add safe.directory /libvgpu && \
            bash ./build.sh"
 .PHONY: build-in-docker

dockerfiles/Dockerfile

@@ -6,7 +6,11 @@ WORKDIR /libvgpu
 COPY . /libvgpu
 
 RUN apt-get update && \
-    apt-get install -y cmake git && \
+    apt-get install -y cmake git libvulkan-dev && \
     rm -rf /var/lib/apt/lists/*
 
 RUN bash ./build.sh
+
+# Ship Vulkan implicit layer manifest so HAMI_VULKAN_ENABLE=1 activates the layer.
+RUN install -D -m 0644 /libvgpu/etc/vulkan/implicit_layer.d/hami.json \
+        /etc/vulkan/implicit_layer.d/hami.json

docs/superpowers/notes/2026-04-28-cuda-hook-audit.md

@@ -0,0 +1,77 @@
+# CUDA hook robustness audit — 2026-04-28
+
+Reference fix: commit `03f99d7 fix(cuda): avoid NULL deref in cuMemGetInfo_v2 when caller (OptiX) crashes`
+
+Pattern (verified against current `cuMemGetInfo_v2` body at `src/cuda/memory.c:501`):
+
+1. Forward to the real driver first (errors surface exactly as without HAMi).
+2. Early return on NULL/invalid args (driver already rejected — never deref).
+3. Then HAMi enforcement / accounting logic.
+
+Why we need this: NVIDIA Isaac Sim Kit (Carbonite / OptiX / Aftermath) calls
+several of these CUDA hooks via internal probes that may pass NULL output
+pointers or run before any CUDA context is current. With the current code the
+HAMi enforcement path runs unconditionally, dereferences the NULL output, or
+calls `cuCtxGetDevice` without checking its return — and `libvgpu.so`
+SegFaults inside the app.
+
+## Hooks needing the same pattern
+
+Line numbers verified against current source on branch `vulkan-layer`
+(`grep -n '^CUresult ' src/cuda/{memory,context}.c` 2026-04-28):
+
+- `cuMemAlloc_v2` — `src/cuda/memory.c:135` (Task 2)
+  - Body: `ENSURE_RUNNING(); allocate_raw(dptr, bytesize)` — `allocate_raw`
+    will write `*dptr` and call into `oom_check` / `add_chunk` without any
+    NULL guard on `dptr`.
+- `cuMemAllocHost_v2` — `src/cuda/memory.c:145` (Task 3)
+  - Body: forwards first (good), but on success does `*hptr = NULL` /
+    re-frees via `*hptr` inside `check_oom` branch without confirming
+    caller passed a non-NULL `hptr`. OOM cleanup path will crash.
+- `cuMemAllocManaged` — `src/cuda/memory.c:159` (Task 3)
+  - Body: `cuCtxGetDevice(&dev)` via `CHECK_DRV_API` — if no current
+    context this aborts; `oom_check` runs before the real driver call so
+    NULL `dptr` isn't surfaced as the driver's own
+    `CUDA_ERROR_INVALID_VALUE`.
+- `cuMemAllocPitch_v2` — `src/cuda/memory.c:174` (Task 4)
+  - Same pattern as Managed: pre-call `cuCtxGetDevice` + `oom_check` then
+    forward. NULL `dptr`/`pPitch` aren't returned by HAMi the way the
+    real driver does, and post-success path writes `*dptr` into
+    `add_chunk_only`.
+- `cuMemHostAlloc` — `src/cuda/memory.c:223` (Task 5)
+  - Forwards first (good), but OOM cleanup writes `*hptr = NULL` without
+    NULL guard on the caller's `hptr` parameter.
+- `cuMemHostRegister_v2` — `src/cuda/memory.c:239` (Task 6)
+  - Calls `cuCtxGetDevice(&dev)` *unconditionally* and ignores its
+    return code (the device variable is then unused — vestigial code).
+    Also runs `check_oom` after success path. Needs forward-first +
+    drop-the-stray-`cuCtxGetDevice` cleanup.
+- `cuCtxGetDevice` — `src/cuda/context.c:42` (Task 7)
+  - Pure passthrough today, but the underlying driver call returns
+    `CUDA_ERROR_INVALID_CONTEXT` when no context is current; several of
+    the hooks above currently rely on `cuCtxGetDevice` succeeding. We
+    add an explicit NULL guard on `device` so HAMi's own callers
+    (which may be called before any real `cuCtxGetCurrent`) don't crash
+    when `device == NULL` is passed during early-init probing.
+
+## Already robust (skip — reference patterns)
+
+- `cuMemFree_v2` — `src/cuda/memory.c:192` (commit `3bebc8a`,
+  "fix(cuda): fall back to real driver on untracked cuMemFree[Async]
+  pointer"): NULL-pointer early return + fall-through to real driver on
+  unknown pointer.
+- `cuMemFreeAsync` — `src/cuda/memory.c:655` (same commit `3bebc8a`).
+- `cuMemGetInfo_v2` — `src/cuda/memory.c:501` (commit `03f99d7`):
+  forward-first + NULL guard + benign return when no current context.
+  This is the canonical reference for Tasks 2–7.
+- `cuMemCreate` — `src/cuda/memory.c:608` (commit `833c62c`,
+  "fix: segfault in cuMemCreate hook when cuCtxGetDevice fails"):
+  guards `cuCtxGetDevice` failure before HAMi enforcement.
+
+## Out-of-scope but noted
+
+- `cuMemFreeHost` / `cuMemHostUnregister` (`memory.c:213`, `:265`) are
+  pure passthroughs and don't need hardening (they only forward to the
+  real driver).
+- `cuMemoryAllocate` (`memory.c:129`) is an internal helper that
+  delegates to `cuMemAlloc_v2` — fixing the latter covers it.

docs/superpowers/notes/2026-04-28-vk-dispatch-lifetime-audit.md

@@ -0,0 +1,119 @@
+# Vulkan dispatch lifetime + chain copy audit (2026-04-28)
+
+Step C Task 5 audit. Read-only review of `src/vulkan/dispatch.c` and `src/vulkan/layer.c` against Vulkan loader spec 1.3 §38.
+
+## Scope
+
+1. Lifetime of `hami_instance_dispatch_t` / `hami_device_dispatch_t` returned by `hami_instance_lookup` / `hami_device_lookup` / `hami_instance_first`.
+2. In-place advance of `chain->u.pLayerInfo` in `hami_vkCreateInstance` / `hami_vkCreateDevice`.
+
+---
+
+## 1. Dispatch lifetime
+
+### Code reviewed
+
+`hami_instance_lookup` (`dispatch.c:49-55`):
+
+```c
+hami_instance_dispatch_t *hami_instance_lookup(VkInstance inst) {
+    pthread_mutex_lock(&g_lock);
+    hami_instance_dispatch_t *p = g_inst_head;
+    while (p && p->handle != inst) p = p->next;
+    pthread_mutex_unlock(&g_lock);   // <-- lock dropped here
+    return p;                         // <-- caller uses p outside lock
+}
+```
+
+Same pattern in `hami_device_lookup` (`dispatch.c:96-102`) and `hami_instance_first` (`dispatch.c:42-47`).
+
+`hami_vkDestroyInstance` (`layer.c:101-106`):
+
+```c
+hami_instance_dispatch_t *d = hami_instance_lookup(instance);
+if (d) d->DestroyInstance(instance, pAllocator);
+hami_instance_unregister(instance);   // frees the node
+```
+
+### Race analysis
+
+**Theoretical race:** Thread A calls `hami_vkDestroyInstance(I)`. Thread B simultaneously calls `vk*` on `I`. Both lookups succeed; Thread A then unregisters and frees the node. Thread B then dereferences a freed dispatch pointer → use-after-free.
+
+**Spec position (Vulkan 1.3 §3.6 "Threading Behavior"):**
+
+> Externally synchronized parameters: The application MUST ensure that no two
+> calls operate on the same handle simultaneously when at least one of them
+> is `vkDestroy*`.
+
+VkInstance, VkDevice, VkQueue, VkCommandBuffer (and command pool) are externally synchronized. The application — not the layer — is responsible for serializing destroy against any concurrent use.
+
+**Real callers:**
+- NVIDIA Carbonite (`libcarb.graphics-vulkan`) destroys VkInstance / VkDevice on a single shutdown thread after stopping all rendering.
+- Isaac Sim Kit follows the same pattern.
+
+**Use of `hami_instance_first()`:**
+Called only from the layer's `vkEnumerateDevice*` hooks (`layer.c:263`, `layer.c:279`), which run during normal lifecycle (after CreateInstance, before DestroyInstance). It does not race with destroy under the spec's external-sync requirement.
+
+### Decision: no code change
+
+The lookup-then-use pattern relies on spec-mandated external synchronization that real-world Vulkan applications satisfy. Adding refcounts or extending the lock across the call would be a behavioral divergence from how every reference layer (Khronos `VK_LAYER_KHRONOS_validation`, `nvidia_layers.json`) handles it — those also drop the lock before invoking the next-chain function pointer, for the same reason: holding a global mutex across an unbounded next-chain call would deadlock.
+
+Documented but not patched. If we ever observe a real crash whose stack matches the use-after-free pattern (Thread B inside a hooked entry point with stale dispatch), revisit with refcounts.
+
+---
+
+## 2. Chain pLayerInfo in-place advance
+
+### Code reviewed
+
+`hami_vkCreateInstance` (`layer.c:75-76`):
+
+```c
+PFN_vkGetInstanceProcAddr next_gipa = chain->u.pLayerInfo->pfnNextGetInstanceProcAddr;
+chain->u.pLayerInfo = chain->u.pLayerInfo->pNext;
+```
+
+Same pattern in `hami_vkCreateDevice` (`layer.c:117-118`).
+
+### Spec / reference layers
+
+**Vulkan-Loader-Interface (`docs/LoaderLayerInterface.md`, Khronos):**
+
+> Layers should follow the same recommendation as drivers and advance the
+> link information before calling down. ... The loader requires that the
+> layer advance the link node so that subsequent layers see the correct
+> next-link.
+
+This is the canonical instance-chain pattern documented in the Khronos vulkan-loader source (`loader/loader.c` and the per-layer examples in `tests/framework/layer/`).
+
+**Reference layer implementations all do in-place advance:**
+
+- `VK_LAYER_KHRONOS_validation` (`layers/state_tracker/instance_state.cpp`):
+  ```cpp
+  chain_info->u.pLayerInfo = chain_info->u.pLayerInfo->pNext;
+  ```
+- NVIDIA's optimus / nvoptix layers (per Khronos sample layers and public layer skeletons): same pattern.
+- Renderdoc's `VK_LAYER_RENDERDOC_Capture`: same pattern.
+
+The in-place advance is the documented recommendation, not a workaround.
+
+### Reuse concern
+
+**Question:** Does NVIDIA driver or Carbonite reuse the same `VkInstanceCreateInfo` after our advance, causing the chain to skip a layer?
+
+**Investigation:** `pCreateInfo` is `const VkInstanceCreateInfo *`; the Vulkan loader allocates a fresh `VkLayerInstanceCreateInfo` per layer per `vkCreateInstance` invocation (see Khronos `vulkan-loader` `loader/trampoline.c::terminator_CreateInstance`). The structure is loader-owned scratch memory, not application memory, so reusing it across calls is not possible — the loader builds a new chain on every call.
+
+**Verified:** within a single `vkCreateInstance` invocation, each layer in the chain advances the same `pLayerInfo` once before calling down, by design. After our advance, the next layer (or driver) sees its own link as the head, not ours. This is exactly the spec contract.
+
+### Decision: no code change
+
+Pattern matches spec recommendation and every reference layer. A deep-copy would diverge from the canonical pattern and add allocation overhead with no behavioral gain.
+
+---
+
+## Conclusion
+
+Both audit areas are clean. Step C does not need additional code patches from this audit. If runtime evidence later contradicts the analysis, the items to look for are:
+
+- **Lifetime:** stack with hooked entry-point on Thread B, freed dispatch on Thread A. Fix path: refcount the dispatch struct or extend the lock with a try-lock + retry.
+- **Chain:** vkCreateDevice receiving a `pLayerInfo` that already points past the next layer. Fix path: deep-copy the `VkLayerDeviceCreateInfo` and patch the copy, restore on return.

docs/superpowers/notes/2026-04-28-vk-trace-isaac-sim.md

@@ -0,0 +1,105 @@
+# Vulkan layer trace — Isaac Sim Kit init under LD_PRELOAD (2026-04-28)
+
+Step C Task 3 trace. **Conclusion: Step C Tasks 1+2 introduce a regression on the LD_PRELOAD-only (no implicit-layer manifest) code path.** Production .so has been restored to the pre-Step-C build pending plan revision.
+
+Build base under test: HAMi-core `vulkan-layer` after Step C Tasks 1+2 (`eea2beb`).
+Build under comparison: HAMi-core `vulkan-layer` Step B end (`7dcb5a4`) — md5 `8f889313ece246b2d08ea6291f48b67a` on `/usr/local/vgpu/libvgpu.so.bak-pre-step-c`.
+New build md5: `9586feee3f0672ab35c4d6f63120dfc4`.
+
+## Methodology
+
+ws-node074 isaac-launchable namespace, pod `isaac-launchable-0-757b765f45-2d76x` (vscode container).
+50s `runheadless.sh` runs with `ACCEPT_EULA=y`. `pkill -KILL kit` between runs.
+
+## Results
+
+### Test matrix (3 isolated runs)
+
+| Run | LD_PRELOAD | implicit layer manifest | exit | crash | listen :49100 | HAMi-core init |
+|---|---|---|---|---|---|---|
+| 1. Baseline | none | none | 124 | 0 | 1 | n/a |
+| 2. Pre-Step-C (`7dcb5a4` backup) | `.so.bak-pre-step-c` | none | 124 | 0 | (alive) | 0 (timeout reached) |
+| 3. Post-Step-C (`eea2beb`) | new `.so` | none | **139** | **2** | (n/a; crashed at 1.5s) | 9 |
+| 4. Post-Step-C + `/tmp/vk-layers/hami.json` (process-scope) | new `.so` | `VK_LAYER_PATH=/tmp/vk-layers VK_INSTANCE_LAYERS=VK_LAYER_HAMI_vgpu` | **139** | **1** | (n/a) | 8 |
+
+Run 4 attempted to enable our layer via process-scope `VK_LAYER_PATH` because writing into `/etc/vulkan/implicit_layer.d/` was sandbox-blocked. **`HAMI_VK_TRACE=1` produced 0 trace lines in Run 4** — the loader did not actually invoke our `vkGetInstanceProcAddr`, suggesting `VK_INSTANCE_LAYERS` is the explicit-layer mechanism and does not apply to NVIDIA's implicit-layer-shaped manifest. We could not get trace evidence under a manifest-activated path in this session.
+
+### HAMI_VK_TRACE lookup names (Run 3 + Run 4)
+
+**0 lines in both.** Vulkan layer init never reached our wrappers — Kit crashed before vkCreateInstance.
+
+### Crash backtrace (Run 3)
+
+```
+000: libc.so.6!__sigaction+0x50
+001: libEGL_nvidia.so.0!__egl_Main+0x3b3
+002: libEGL_nvidia.so.0!__egl_Main+0x1a27
+003: libGLX_nvidia.so.0!vk_icdNegotiateLoaderICDInterfaceVersion+0x3b9
+004: libGLX_nvidia.so.0!__glx_Main+0x2b2d
+005: libGLX_nvidia.so.0!vk_icdNegotiateLoaderICDInterfaceVersion+0x12
+006: libvulkan.so.1!+0x31724
+007: libvulkan.so.1!+0x32033
+008: libvulkan.so.1!+0x31db0
+```
+
+NVIDIA driver 580.142, RTX 6000 Ada Generation, 46068 MiB.
+
+This is the Vulkan loader (`libvulkan.so.1`) loading the NVIDIA ICD (`libGLX_nvidia.so.0`'s `vk_icdNegotiateLoaderICDInterfaceVersion`), which dispatches into NVIDIA's EGL backend (`libEGL_nvidia.so.0!__egl_Main`), which crashes inside `__sigaction` setup. The crash is during ICD initialization — before any `vkCreateInstance` could have run, so our layer's `g_first_next_gipa` was still NULL.
+
+## Hypothesis (root cause)
+
+LD_PRELOAD-only path (no implicit-layer manifest):
+
+1. Our `libvgpu.so` exports `vkGetInstanceProcAddr` and `vkGetDeviceProcAddr` with default visibility — these symbols enter the global `RTLD_DEFAULT` namespace at process start.
+2. NVIDIA's ICD (`libGLX_nvidia.so.0`) initializes through `libvulkan.so.1` and resolves several Vulkan entry points by name. Some of those resolutions reach our exported symbols instead of the loader-routed ICD entry, because we are LD_PRELOAD'd ahead.
+3. Step B's `7dcb5a4` exported the same wrappers but with a NULL-on-unknown-instance fallback for `GIPA`/`GDPA`, so those paths returned NULL → ICD treated as "function not present" → ICD self-resolved or short-circuited gracefully.
+4. Step C Tasks 1+2 changed two things on top of that:
+   - **Task 1:** added `HAMI_HOOK(EnumerateDeviceExtensionProperties)` and `HAMI_HOOK(EnumerateDeviceLayerProperties)` — meaning a global GIPA lookup for those names now returns our wrapper instead of falling through.
+   - **Task 2:** changed unknown-instance fallback in `hami_vkGetInstanceProcAddr` / `hami_vkGetDeviceProcAddr` from "return NULL" to "forward via cached `g_first_next_gipa`".
+5. Under LD_PRELOAD-only, our `g_first_next_gipa` is NULL during ICD bring-up because `vkCreateInstance` has not yet run. So Task 2's fallback collapses back to the original NULL return — it should be no worse.
+6. The remaining suspect is Task 1: the new `EnumerateDeviceExtensionProperties` wrapper, when `g_inst_head == NULL`, returns `pPropertyCount=0, VK_SUCCESS`. NVIDIA's ICD/EGL backend may consult device extensions during EGL bringup and treat 0 entries as a hard error, or NULL-deref a result pointer it expected to be populated. Pre-Step-C had no hook for that name, so the GIPA chain returned NULL → NVIDIA fell back to its own internal table.
+
+This is a hypothesis from comparative evidence + backtrace, not from a working trace. The smoking-gun trace would be a HAMI_VK_TRACE log under a properly-activated implicit-layer manifest, which the current sandbox prevents.
+
+## Decision (Task 4)
+
+**Stop and surface to controller.** The Plan's Task 4 was framed as "add hooks for additional vkGetPhysicalDevice* names that returned NULL in trace" — an additive change. The actual evidence calls for the opposite: **revisit Tasks 1+2 because they introduced a runtime regression on the LD_PRELOAD-only path, which is the path Plan Task 6 explicitly verifies.**
+
+Options for the controller:
+
+1. **Revert Task 1's EnumerateDevice* hooks for the unknown-instance case** — return NULL (pre-Step-C behavior) when `g_inst_head == NULL`, and only forward to `hami_instance_first()` when an instance is registered. Preserves the Carbonite-fix intent for the manifest path while staying inert on LD_PRELOAD-only.
+2. **Gate the layer's wrapper exports on a HAMI-mode check** — at .so init, detect whether our layer manifest is active (e.g., via `VK_LAYER_HAMI_vgpu` in `VK_INSTANCE_LAYERS` or environment) and turn the GIPA/GDPA + Enumerate hooks into pass-throughs otherwise.
+3. **Dlsym(RTLD_NEXT) fallback** — when `g_first_next_gipa == NULL`, resolve `vkGetInstanceProcAddr` via `RTLD_NEXT` and forward, so ICD init via LD_PRELOAD never sees our hooks return surprising values.
+
+Option 1 is the smallest delta and most likely to keep the Step B regression-pass intact.
+
+## Update (Run 5): hypothesis falsified
+
+A Task 1-targeted fix was attempted: `HAMI_HOOK_GATED(EnumerateDeviceExtensionProperties)` and `HAMI_HOOK_GATED(EnumerateDeviceLayerProperties)` — return our wrapper only when `hami_instance_first() != NULL`, fall through to NULL otherwise.
+
+| Run | Build | exit | crash | listen | trace lines |
+|---|---|---|---|---|---|
+| 5. Step C + gated hooks (md5 1048daaf) | new | **139** | **2** | 0 | **0** |
+
+Same crash, same backtrace, same `HAMI_VK_TRACE=0 lines`. The gate addressed the only theory we had for how Tasks 1+2 could affect ICD init, and it changed nothing.
+
+**Conclusion:** the regression is NOT triggered by NVIDIA ICD calling our `vkGetInstanceProcAddr`. Our wrapper is genuinely never called — `HAMi-core init=9` confirms `LD_PRELOAD` succeeded but the Vulkan loader code path that would call our GIPA hasn't run by the time of the crash.
+
+Yet the new build crashes and the old build (`8f889313`) does not. So the differential is somewhere that runs **at .so load time or earlier in NVIDIA driver init**, not in our Vulkan wrappers. Candidate diff surface:
+
+- ELF symbol exports added by Step C (e.g., `hami_instance_first` is now a non-static external) — could collide with a NVIDIA driver weak symbol or change global lookup order.
+- Static initializer / constructor side effects from new TUs being linked in (`dispatch.c` now compiles slightly more code).
+- Any change between `8f889313`'s source state and `7dcb5a4` that we have not yet identified — the backup md5 was created before this session and may NOT correspond to commit `7dcb5a4`. We cannot bisect further without rebuilding `7dcb5a4` cleanly on ws-node074, which the next attempt was sandbox-blocked.
+
+## Diagnostics not yet possible in this session
+
+- Build commit `7dcb5a4` cleanly and md5-compare to `8f889313` — would confirm whether prod backup is from Step B end or some earlier commit.
+- `nm -D libvgpu.so` symbol diff between `8f889313` and `9586feee` — would reveal new exports.
+- `LD_DEBUG=symbols,bindings` under runheadless — would show which symbol resolution actually picks up our `.so` first.
+
+## Cleanup performed (this session)
+
+- Restored `/usr/local/vgpu/libvgpu.so` from `.bak-pre-step-c` on ws-node074. md5 `8f889313ece246b2d08ea6291f48b67a` confirmed (3x: after Run 3, after Run 4, after Run 5).
+- Removed pod `/tmp/vk-layers` and `/tmp/vk-trace`.
+- Confirmed isaac-launchable-0 baseline still alive after each restore: `exit=124 crash=0 listen=1`.
+- Discarded the unproven gate edit from `src/vulkan/layer.c`.