Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving

An Efficient and Versatile Inference Engine for Distributed LLM Serving

A deterministic PyTorch autograd verification trap for catching silent KV-cache routing and block-alignment failures in vLLM and SGLang serving infrastructure.

Deadline-aware KV-cache scheduling for protecting decode-critical request-state under long-context LLM inference pressure.

🚀 Accelerate LLM inference with Mini-Infer, a high-performance engine designed for efficiency and power in AI model deployment.

What to consider when running AI Inference at scale on Kubernetes

A minimal LLM inference engine implementing PagedAttention-style KV cache management on NanoGPT. Based on the "Efficient Memory Management for Large Language Model Serving with PagedAttention" paper.

From-scratch model of an LLM serving engine's systems core: paged KV-cache, continuous batching, preemption, and prefix caching — GPU-free, with reproducible benchmarks.