Dynamic Batching Inference System

A simple inference server built with FastAPI that demonstrates dynamic batching and dynamic padding for Transformer model inference.

Project Overview

Modern inference servers improve GPU and CPU utilization by processing multiple user requests together instead of executing every request immediately.

This project demonstrates:

• Dynamic batching using an asynchronous request queue.
• Dynamic padding using the Hugging Face tokenizer.
• A FastAPI inference API.
• A benchmark script for measuring throughput and latency.

Project Structure

dynamic-batching-inference-system/
│
├── app/
│   ├── main.py          # FastAPI server
│   ├── batcher.py       # Dynamic batching logic
│   └── model.py         # Transformer model wrapper
│
├── benchmark/
│   └── load_test.py     # Load testing script
│
├── requirements.txt
└── README.md

Dynamic Batching

Incoming requests are placed into an asynchronous queue.

The batcher waits for a small time window (20 ms by default), groups multiple requests together, and executes one model inference for the entire batch.

This reduces the number of forward passes and improves throughput.

Dynamic Padding

Before inference, all requests are tokenized together.

Instead of padding every sequence to a fixed maximum length, each batch is padded only to the longest sequence inside that batch.

Example:

Original token lengths: [4, 31, 111]
Padded batch length: 111

This reduces unnecessary computation compared with static padding.

Running the Project

Install dependencies:

pip install -r requirements.txt

Start the API:

uvicorn app.main:app --reload

Test the API

curl -X POST http://127.0.0.1:8000/predict \
-H "Content-Type: application/json" \
-d '{"text":"Dynamic batching is useful"}'

Benchmark

Run the comparison benchmark:

python benchmark/load_test.py

This script now compares:

• Dynamic batching against direct inference.
• Dynamic batching against static batching.
• Batch-size sensitivity.
• Queue-wait-time sensitivity.
• Dynamic padding against fixed-length padding.

It prints a short summary and writes SVG charts to the benchmark plots folder:

• benchmark/plots/batching_comparison.svg
• benchmark/plots/batch_size_comparison.svg
• benchmark/plots/queue_wait_comparison.svg
• benchmark/plots/batching_strategy_comparison.svg
• benchmark/plots/padding_comparison.svg

Benchmark Graphs

Batch size impact

This graph shows how throughput changes as batch size increases. Larger batches generally improve throughput, but they also increase the maximum amount of data processed per forward pass.

Queue wait time impact

This graph compares different dynamic batching wait windows (10 ms, 20 ms, 50 ms). Longer waits allow more requests to accumulate in a batch, which can improve throughput at the cost of higher latency.

Dynamic vs static batching

This chart compares adaptive dynamic batching with a simple static batch strategy. Dynamic batching is more responsive to varying request arrival rates, while static batching may be more efficient only when request arrival is highly regular.

Throughput comparison

This comparison shows direct single-request inference versus the dynamic batching approach. Dynamic batching typically improves throughput by combining multiple requests into a single model invocation.

Padding cost comparison

This plot compares the effective token cost of dynamic padding versus fixed-length padding. Dynamic padding saves computation by only padding each batch to its longest sequence, instead of using a larger fixed length for every batch.

Example benchmark summary:

Dynamic batching vs direct inference
- Direct inference throughput: 666.67 requests/s
- Dynamic batching throughput: 1000.00 requests/s
- Direct average latency: 0.0015 s
- Dynamic average latency: 0.0200 s

Dynamic padding vs fixed-length padding
- Dynamic padded length: 111
- Fixed padded length: 128
- Token cost reduction: 288
- Token cost ratio: 0.87

Comparison Notes

Dynamic batching usually improves throughput because several requests can share one forward pass, but it can add a small queueing delay while requests wait for the batch window to close.

Dynamic padding improves efficiency because the batch is only padded to the longest sequence in that batch, rather than to a larger fixed maximum length used by every batch.

Scheduler Strategies

This project includes an extensible scheduler interface under app/schedulers/ with two example implementations:

• FIFOScheduler — preserves request arrival order.
• LengthAwareScheduler — orders requests by token length (longest first) to reduce padding waste inside a batch.

You can compare the schedulers with the helper script:

python benchmark/compare_schedulers.py --requests 1000 --batch-size 8

The script writes a brief summary to benchmark/plots/scheduler_summary.txt showing total batch token cost for each scheduler and the relative reduction.

Extended usage

The comparison script now supports running multiple random seeds and batch sizes, and will write aggregated CSV and SVG plot outputs to benchmark/plots/.

Example:

# run with three batch sizes and three random seeds
python benchmark/compare_schedulers.py --requests 1000 --batch-sizes 4 8 16 --seeds 0 1 2 --output-dir benchmark/plots

Files produced

• benchmark/plots/scheduler_results.csv — per-run CSV with tokens for each scheduler
• benchmark/plots/scheduler_fifo_by_batch.svg — aggregated FIFO token cost by batch size (SVG)
• benchmark/plots/scheduler_length_by_batch.svg — aggregated length-aware token cost by batch size (SVG)
• benchmark/plots/scheduler_grouped_by_batch.svg — aggregated grouped-by-length token cost by batch size (SVG)

Note: the LengthAwareScheduler implemented here only reorders requests inside an existing batch (it does not change which requests are grouped together). To reduce padding most effectively, grouping similar-length requests into the same batches (shown by the "grouped" chart) is usually required.

Interpretation

Compare the two SVGs or inspect the CSV to see which scheduler reduces padding/token cost for your workload. Try varying --requests, --batch-sizes, and --seeds to explore different traffic patterns.

Scheduler Figures (embedded)

Figure 1. FIFO scheduler total batch token cost by batch size. Lower is better. (High-res: SVG.)

Figure 2. Length-aware ordering (descending length inside each batch). This reordering helps within-batch padding but does not change which requests are grouped together. (High-res: SVG.)

Figure 3. Grouped-by-length (bucketing similar lengths before batching) — typically yields the largest reduction in padding cost. (High-res: SVG.)

Note: images are sized for README readability; the original SVG files in benchmark/plots/ are available for high-resolution inspection.

Scheduler sweep summary

We ran a parameter sweep varying the fraction of short requests (0.1 → 0.9) and aggregated results across multiple random seeds. Grouping by length (bucketing similar-length requests before batching) consistently reduced total token work compared to FIFO/length-aware ordering — and the benefit increases as workloads become more clustered (many short requests with fewer long ones).

Key artifacts:

• CSV of raw results: benchmark/plots/scheduler_sweep.csv
• Plots (percent reduction of Grouped vs FIFO):
  • batch=8
  • batch=16
  • batch=32

Figure: Mean percent reduction (Grouped vs FIFO) across short-frac values for batch=8. Higher values indicate a larger reduction in padding waste when grouping by length.

short_frac	batch=8	batch=16	batch=32
0.1	22.9%	24.0%	24.1%
0.3	37.2%	38.5%	38.8%
0.5	51.1%	53.6%	54.0%
0.7	63.4%	67.9%	69.2%
0.9	70.1%	78.7%	82.9%

Future Improvements

• Add GPU support.
• Support larger Transformer models.
• Measure these comparisons against a real deployed server.