Skip to content

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

prefill-decode-interference-bench

Python PyTorch HuggingFace CUDA License Status

Benchmark measuring how much prefill and decode hurt each other when co-located on the same GPU, with a focus on TTFT inflation experienced by new requests arriving on a busy server.


Why This Exists

disaggregated-prefill-decode-sim showed a 10.76x throughput gain from disaggregation via an analytical model. But that model did not measure actual GPU interference between phases.

This benchmark closes that loop with real hardware measurements.

The central question:

When prefill and decode run simultaneously on the same GPU, how much does a new request's TTFT increase?


Key Results

TTFT inflation is the primary cost of co-location

Qwen2-0.5B stable results (median across 3 examples):

prompt_len batch cold TTFT hot TTFT TTFT inflation penalty
128 1 25.4ms 31.0ms 1.22x +5.6ms
128 4 25.1ms 30.1ms 1.23x +5.8ms
256 1 32.1ms 38.3ms 1.19x +6.2ms
512 1 50.7ms 63.7ms 1.27x +13.7ms
512 4 47.1ms 58.8ms 1.25x +11.9ms

The TTFT penalty grows with prompt length. At 512 tokens, a new request waits 12-14ms extra because a decode loop is running on the same GPU.

Decode throughput is nearly unaffected

decode_slowdown across all 0.5B scenarios: 0.95x to 1.07x

Decode is memory-bandwidth-bound. Prefill is compute-bound. They compete for different resources, so decode is largely immune.

Larger models show worse TTFT pollution

Qwen2-1.5B worst cases:

prompt_len batch TTFT inflation penalty
128 4 1.48x +17ms
256 1 1.74x +36ms
512 4 4.96x +420ms

Co-location does not produce real parallelism

wall_overlap_efficiency ≈ 0.91–1.02x across all 0.5B scenarios.

A perfect parallel execution would give ~2x efficiency. Instead, co-location is essentially serial execution with TTFT inflation.


Measurement Method

Three measurement modes per configuration:

cold_prefill — GPU idle baseline

hot_prefill — prefill fired while decode loop runs on second model replica. This directly measures the TTFT seen by a new user on a busy server.

parallel — both dispatched simultaneously on separate CUDA streams, measuring each independently plus total wall time.

Three corpus examples per (prompt_len, batch_size). Median is the primary reported metric.


Quick Start

git clone https://github.com/JohnScheuer/prefill-decode-interference-bench
cd prefill-decode-interference-bench

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

python run.py

Requires approximately 4-8 GB VRAM depending on model. Runtime: approximately 10-20 minutes per model on RTX 2070.


Output Files

results/
  interference.csv   per-example raw measurements
  summary.csv        aggregated by model, prompt_len, batch_size

plots/
  01_ttft_inflation.png          TTFT inflation vs prompt_len
  02_ttft_penalty_ms.png         Absolute TTFT penalty in ms
  03_decode_slowdown_secondary.png   Decode slowdown (near 1.0)
  04_wall_overlap_efficiency.png     Co-located wall efficiency

Project Structure

prefill-decode-interference-bench/
├── src/
│   ├── config.py        models, prompt lengths, batch sizes
│   ├── data.py          corpus example builder
│   ├── runner.py        cold, hot, and parallel measurement methods
│   ├── bench.py         sweep and aggregation
│   └── analysis.py      plots and summary tables
├── results/
├── plots/
├── run.py
├── SUMMARY.txt
├── DESIGN.md
├── LICENSE
└── requirements.txt

Interpretation Note

TTFT inflation values below 1.0 should be treated as measurement noise floor. The GPU scheduler introduces some jitter that can make isolated measurements appear slightly faster or slower. The signal is in values consistently above 1.0, especially at longer prompt lengths.


Requirements

  • Python 3.10+
  • PyTorch >= 2.1.0
  • Transformers >= 5.0.0
  • Pandas >= 2.0.0
  • Matplotlib >= 3.8.0
  • NumPy >= 1.26.0
  • NVIDIA GPU with >= 6GB VRAM (8GB recommended)

What Was Not Measured

  • Sustained load with many concurrent requests
  • Decode batch sizes above 4 on 1.5B (OOM risk)
  • Prompt lengths above 512 tokens
  • Models larger than 1.5B
  • Effect of KV cache pressure from co-location

Documentation

  • DESIGN.md — full design rationale and module descriptions
  • SUMMARY.txt — plain-text findings with all numbers
  • LICENSE — MIT License

Related Projects


License

MIT License — Copyright (c) 2026 João Felipe De Souza

See LICENSE for details.


Author

João Felipe De Souza

About

Benchmark for prefill/decode interference on a single GPU, measuring TTFT inflation when new requests arrive while decode is already running.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages