Benchmark measuring how much prefill and decode hurt each other when co-located on the same GPU, with a focus on TTFT inflation experienced by new requests arriving on a busy server.
disaggregated-prefill-decode-sim showed a 10.76x throughput gain from disaggregation via an analytical model. But that model did not measure actual GPU interference between phases.
This benchmark closes that loop with real hardware measurements.
The central question:
When prefill and decode run simultaneously on the same GPU, how much does a new request's TTFT increase?
Qwen2-0.5B stable results (median across 3 examples):
| prompt_len | batch | cold TTFT | hot TTFT | TTFT inflation | penalty |
|---|---|---|---|---|---|
| 128 | 1 | 25.4ms | 31.0ms | 1.22x | +5.6ms |
| 128 | 4 | 25.1ms | 30.1ms | 1.23x | +5.8ms |
| 256 | 1 | 32.1ms | 38.3ms | 1.19x | +6.2ms |
| 512 | 1 | 50.7ms | 63.7ms | 1.27x | +13.7ms |
| 512 | 4 | 47.1ms | 58.8ms | 1.25x | +11.9ms |
The TTFT penalty grows with prompt length. At 512 tokens, a new request waits 12-14ms extra because a decode loop is running on the same GPU.
decode_slowdown across all 0.5B scenarios: 0.95x to 1.07x
Decode is memory-bandwidth-bound. Prefill is compute-bound. They compete for different resources, so decode is largely immune.
Qwen2-1.5B worst cases:
| prompt_len | batch | TTFT inflation | penalty |
|---|---|---|---|
| 128 | 4 | 1.48x | +17ms |
| 256 | 1 | 1.74x | +36ms |
| 512 | 4 | 4.96x | +420ms |
wall_overlap_efficiency ≈ 0.91–1.02x across all 0.5B scenarios.
A perfect parallel execution would give ~2x efficiency. Instead, co-location is essentially serial execution with TTFT inflation.
Three measurement modes per configuration:
cold_prefill — GPU idle baseline
hot_prefill — prefill fired while decode loop runs on second model replica. This directly measures the TTFT seen by a new user on a busy server.
parallel — both dispatched simultaneously on separate CUDA streams, measuring each independently plus total wall time.
Three corpus examples per (prompt_len, batch_size). Median is the primary reported metric.
git clone https://github.com/JohnScheuer/prefill-decode-interference-bench
cd prefill-decode-interference-bench
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python run.py
Requires approximately 4-8 GB VRAM depending on model. Runtime: approximately 10-20 minutes per model on RTX 2070.
results/
interference.csv per-example raw measurements
summary.csv aggregated by model, prompt_len, batch_size
plots/
01_ttft_inflation.png TTFT inflation vs prompt_len
02_ttft_penalty_ms.png Absolute TTFT penalty in ms
03_decode_slowdown_secondary.png Decode slowdown (near 1.0)
04_wall_overlap_efficiency.png Co-located wall efficiency
prefill-decode-interference-bench/
├── src/
│ ├── config.py models, prompt lengths, batch sizes
│ ├── data.py corpus example builder
│ ├── runner.py cold, hot, and parallel measurement methods
│ ├── bench.py sweep and aggregation
│ └── analysis.py plots and summary tables
├── results/
├── plots/
├── run.py
├── SUMMARY.txt
├── DESIGN.md
├── LICENSE
└── requirements.txt
TTFT inflation values below 1.0 should be treated as measurement noise floor. The GPU scheduler introduces some jitter that can make isolated measurements appear slightly faster or slower. The signal is in values consistently above 1.0, especially at longer prompt lengths.
- Python 3.10+
- PyTorch >= 2.1.0
- Transformers >= 5.0.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
- NumPy >= 1.26.0
- NVIDIA GPU with >= 6GB VRAM (8GB recommended)
- Sustained load with many concurrent requests
- Decode batch sizes above 4 on 1.5B (OOM risk)
- Prompt lengths above 512 tokens
- Models larger than 1.5B
- Effect of KV cache pressure from co-location
- DESIGN.md — full design rationale and module descriptions
- SUMMARY.txt — plain-text findings with all numbers
- LICENSE — MIT License
- disaggregated-prefill-decode-sim
- inference-latency-breakdown
- decode-batching-profiler
- continuous-batching-fragmentation-sim
MIT License — Copyright (c) 2026 João Felipe De Souza
See LICENSE for details.
João Felipe De Souza