Buffer pool with liveness-based reuse to reduce peak VRAM usage

[Copilot]

All intermediate buffers were allocated upfront with no reuse — peak VRAM equals the sum of all intermediate buffers, not just those live simultaneously. This causes OOM on 8GB devices.

Changes

• Replace per-node allocation with a buffer pool: buffers: FxHashMap<NodeIndex, CudaSlice<u8>> → buffer_pool: Vec<CudaSlice<u8>> + buffer_node_to_pool: FxHashMap<NodeIndex, usize>. Multiple nodes with non-overlapping lifetimes share a single pool entry.

• Liveness analysis on exec_graph topology: Compute [birth, death) intervals for each buffer based on which exec_graph nodes reference it (inputs, output, extra_buffer_nodes). Buffers internal to the same megakernel correctly get overlapping ranges and are never shared.

• Greedy best-fit allocation: Process nodes in birth order. When a pool entry's last consumer precedes the current node's birth, it enters the free list. New allocations first check for the smallest reusable buffer ≥ required size.

• Zeroing efficiency: Loop iterates over unique pool entries instead of all node mappings.

// Before: one allocation per node, all live simultaneously
self.buffers.insert(node, self.cuda_stream.alloc_zeros(size).unwrap());

// After: reuse pool entries across non-overlapping lifetimes
let pool_idx = if let Some((free_idx, _)) = best_fit {
    free_list.swap_remove(free_idx).1
} else {
    let idx = self.buffer_pool.len();
    self.buffer_pool.push(self.cuda_stream.alloc_zeros(size).unwrap());
    idx
};
self.buffer_node_to_pool.insert(node, pool_idx);

For typical transformer inference, many activation buffers are short-lived (produced then consumed by the next layer), so pool reuse can reduce peak intermediate VRAM substantially.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/asglover/candle/commits/fix/disable-bf16-wmma-pre-ampere
  • Triggering command: /home/REDACTED/.rustup/toolchains/stable-x86_64-REDACTED-linux-gnu/bin/cargo /home/REDACTED/.rustup/toolchains/stable-x86_64-REDACTED-linux-gnu/bin/cargo check -p luminal_cuda (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.