Skip to content

README.md

Latest commit

 

History

History
149 lines (98 loc) · 4.87 KB

README.md

File metadata and controls

149 lines (98 loc) · 4.87 KB

🧪 Python Benchmark Suite

This repository provides a reproducible framework for benchmarking Python language behavior and runtime performance across multiple interpreter versions managed via pyenv.

It is designed for senior engineers and performance-minded developers who need to quantify differences in execution cost (e.g., attribute access, function call overhead, dataclass vs. property performance, etc.) between Python releases.

⚙️ Features

  • 🔍 Version-to-version comparison — run the same benchmark across multiple Python versions.
  • 🧩 Automatic benchmark discovery — any file inside benchmarks/ ending with _benchmark.py is picked up automatically.
  • 🧱 Isolated execution — each benchmark runs independently for every interpreter.
  • 📈 Structured output — generates Markdown summary tables across Python versions and approaches.
  • 🧰 Template generator — create new benchmarks quickly using ./new_benchmark.sh.

📂 Repository Structure

python_benchmark/
├── run.sh             # Benchmark orchestrator
├── new_benchmark.sh   # Template generator for new benchmarks
├── benchmarks/        # Your benchmark scripts (*.py)
└── results/           # Auto-generated benchmark results (gitignored)

Each Python script placed inside benchmarks/ with the suffix _benchmark.py will be automatically discovered and executed by run.sh.

You can implement any benchmark you want — whether testing a micro-optimization, comparing architectural patterns, or evaluating performance between frameworks. As long as the file prints labeled results (e.g., Approach A: <time>), it will be included in the generated summary tables.

🧰 Requirements

  • Linux / macOS
  • pyenv installed and configured 
    curl https://pyenv.run | bash
  • Install a few Python versions to test: 
    pyenv install 3.9.21
pyenv install 3.10.14
pyenv install 3.12.11
pyenv install 3.13.4
pyenv install 3.14.0

🚀 Running Benchmarks

To run all benchmarks sequentially across all Python versions:

./run.sh

To run benchmarks and generate a Markdown summary report:

./run.sh --summary-table

This will produce a file at:

results/summary.md

Each benchmark produces its own section in that file.

🧩 Creating a New Benchmark

You can generate a new benchmark template quickly:

./new_benchmark.sh dict_vs_list

This creates a file at benchmarks/dict_vs_list_benchmark.py pre-populated with a minimal benchmark template using Python’s timeit module.

After generation, you can modify it to benchmark whatever behavior, library, or architectural pattern you want to measure — for example:

  • Comparing direct vs. property access.
  • Comparing dataclass instantiation vs. manual __init__.
  • Comparing service-layer orchestration vs. direct domain calls.

As long as your benchmark prints labeled times in the format:

Approach Name: <seconds>

the results will appear correctly in the final summary.

📊 Example Results

When you run the suite with --summary-table, all results are saved to:

results/summary.md

Each benchmark script produces a separate Markdown table showing all Python versions as rows and each tested approach as a column.

Example 1 — property_benchmark.py

Python Version Property Access (s) Direct Access (s)
3.9.21 0.287390 0.877322
3.14.0 0.126226 0.283366

➡️ In this benchmark, Property Access shows a ~3× speed improvement between Python 3.9 and 3.14, indicating significant runtime optimizations in attribute handling.

Example 2 — json_vs_pydantic_validation_benchmark.py

Python Version Dataclass Validation (s) JSON Parse Only (s) Pydantic Validation (s)
3.9.21 0.210205 0.286122 0.132171
3.14.0 0.184083 0.238300 0.128887

➡️ Here, both Dataclass Validation and Pydantic Validation exhibit minor runtime gains, while JSON parsing overhead remains relatively stable across versions.

🧠 Design Philosophy

This project embodies clarity, reproducibility, and isolation:

  • Every benchmark runs independently of global state.
  • pyenv ensures consistent interpreter environments.
  • Bash automation allows exact replication of runs across machines.
  • Dependencies are managed reproducibly via uv and pyproject.toml.

🧾 License

This project is licensed under the MIT License.
See LICENSE for details.