Skip to content

README.md

Latest commit

 

History

History
376 lines (282 loc) · 17.4 KB

README.md

File metadata and controls

376 lines (282 loc) · 17.4 KB

bench-pg

License

bench-pg is a cross‑language PostgreSQL benchmarking suite designed to measure and compare the performance of different database driver libraries under a realistic, business‑like workload. It provides three independent implementations:

  • Rust – bench-pg-rs (libraries: tokio-postgres, sqlx, diesel)
  • Go – bench-pg-go (libraries: gorm, go-sqlx)
  • Java – bench-pg-java (libraries: spring-jpa, spring-jdbc, raw-jdbc, jooq) – vibecoded using Claude
  • Scala – bench-pg-scala (libraries: slick, zio-quill) – vibecoded using Claude

All tools generate an identical dataset (customers, vendors, materials, users) and execute a configurable number of transactional operations over a simulated time period. This allows you to directly compare throughput and latency across languages and database abstraction layers under exactly the same workload.

Features

  • Multi‑language, multi‑library – Compare Rust, Go, Java, and Scala across different ORMs and drivers.
  • Configurable dataset size – Control the number of customers, vendors, materials, and users.
  • Adjustable workload – Set total operations, time span (years), and connection pool size.
  • Automated schema management – Tables are dropped and recreated on each run.
  • Consistent CLI – Nearly identical command‑line options across all implementations.
  • Virtual Threads support – Java implementation supports Java 21 virtual threads for lightweight concurrency.
  • Descriptive benchmark naming – Auto‑generated or user‑supplied names for easy identification.

Requirements

  • PostgreSQL 15+ (any recent version)
  • Rust (edition 2024) – only for bench-pg-rs
  • Go 1.25+ – only for bench-pg-go
  • Java 21+ with Maven – only for bench-pg-java
  • Scala 3.6+ with sbt 1.10+ – only for bench-pg-scala
  • A PostgreSQL database with a user that has full read/write and schema modification privileges.

Repository Structure

bench-pg/
├── golang/             # Go implementation (bench-pg-go)
│   ├── go.mod
│   ├── main.go
│   └── ...
├── java/               # Java implementation (bench-pg-java)
│   ├── pom.xml
│   ├── src/
│   └── ...
├── rust/               # Rust implementation (bench-pg-rs)
│   ├── Cargo.toml
│   ├── src/
│   └── ...
├── scala/              # Scala implementation (bench-pg-scala)
│   ├── build.sbt
│   ├── project/
│   ├── src/
│   └── ...
├── sql/               
│   └── ddl.sql         # Database schema
└── README.md           # This file

Database

Database should be created and schema must be initialized. Schema file is located in sql/ddl.sql

Rust Application (bench-pg-rs)

The Rust benchmark is located in the rust/ directory. It supports three database libraries via the --lib flag.

Building

cd rust
cargo build --release

The executable will be available at ./target/release/bench-pg-rs.

Command Line Options

Short Long Description Default Possible Values
-U --username PostgreSQL username postgres
-P --password PostgreSQL password postgres
-H --host PostgreSQL host localhost
-p --port PostgreSQL port 5432
-C --connections Size of the connection pool 40
-d --db Database name benchmark
-l --lib Database library to use sqlx tokio, sqlx, diesel
-c --customers Number of customers to generate 100
-v --vendors Number of vendors to generate 100
-m --materials Number of materials to generate 100
-u --users Number of system users 40
-s --start-year Start year for simulated operations 2025
-y --years Number of years of activity 1
-o --operations Total number of operations to execute 20000
-n --name Custom name for the benchmark run (auto‑gen)
-h --help Print help

Example

./target/release/bench-pg-rs -l sqlx -C 50 -o 50000 -n "sqlx-50conn-50kops"

Go Application (bench-pg-go)

The Go benchmark is located in the golang/ directory. It supports two database libraries via the --lib flag.

Building

cd golang
go build -o bench-pg-go .

The executable will be created as ./bench-pg-go.

Command Line Options

All options are passed as long flags (e.g., --host localhost).

Flag Description Default Possible Values
--username PostgreSQL username postgres
--password PostgreSQL password postgres
--host PostgreSQL host localhost
--port PostgreSQL port 5432
--connections Size of the connection pool 40
--db Database name benchmark
--lib Database library to use go-sqlx gorm, go-sqlx
--customers Number of customers to generate 100
--vendors Number of vendors to generate 100
--materials Number of materials to generate 100
--users Number of system users 40
--start-year Start year for simulated operations 2025
--years Number of years of activity 1
--operations Total number of operations to execute 20000
--name Custom name for the benchmark run (auto‑gen)

Example

./bench-pg-go --lib gorm --connections 30 --operations 10000 --name "gorm-30conn-10kops"

Java Application (bench-pg-java)

The Java benchmark is located in the java/ directory. It is built with Spring Boot 3.3 and Java 21 and supports three database libraries via the --lib flag:

  • spring-jpa – Spring Data JPA (Hibernate) with full entity lifecycle management
  • spring-jdbc – Spring JDBC (NamedParameterJdbcTemplate), lighter than JPA
  • raw-jdbc – Direct JDBC (PreparedStatement), no Spring ORM overhead, manual transaction control
  • jooq – jOOQ type-safe SQL DSL with code generation from the DDL schema

The application also supports Java 21 Virtual Threads (--virtual-threads true), which enables lightweight concurrency without OS-thread-per-user overhead.

Building

cd java
mvn package -DskipTests

The executable JAR will be available at ./target/bench-pg-java.jar.

Command Line Options

All options are passed as long flags (e.g., --host localhost).

Flag Description Default Possible Values
--username PostgreSQL username postgres
--password PostgreSQL password postgres
--host PostgreSQL host localhost
--port PostgreSQL port 5432
--connections Size of the connection pool 40
--db Database name benchmark
--lib Database library to use raw-jdbc spring-jpa, spring-jdbc, raw-jdbc, jooq
--customers Number of customers to generate 100
--vendors Number of vendors to generate 100
--materials Number of materials to generate 100
--users Number of concurrent worker threads 20
--start-year Start year for simulated operations 2025
--years Number of years of activity 1
--operations Total number of operations to execute 15000
--name Custom name for the benchmark run (auto‑gen)
--virtual-threads Use Java 21 virtual threads instead of OS threads true true, false

Example

java -jar target/bench-pg-java.jar --lib raw-jdbc --connections 40 --users 20 --operations 15000
java -jar target/bench-pg-java.jar --lib spring-jpa --virtual-threads false --name "jpa-os-threads"

Scala Application (bench-pg-scala)

The Scala benchmark is located in the scala/ directory. It is built with Scala 3.6 and sbt and supports two database libraries via the --lib flag:

  • slick – Slick 3.5, async DBIO actions, FOR UPDATE via raw SQL interpolation
  • zio-quill – ZIO Quill 4.8, functional SQL DSL running on ZIO, bridged to synchronous API

Building

cd scala
sbt assembly

The fat JAR will be available at ./target/scala-3.6.4/bench-pg-scala.jar.

Command Line Options

All options are passed as long flags (e.g., --host localhost).

Flag Description Default Possible Values
--username PostgreSQL username postgres
--password PostgreSQL password postgres
--host PostgreSQL host localhost
--port PostgreSQL port 5432
--connections Size of the connection pool 40
--db Database name benchmark
--lib Database library to use zio-quill slick, zio-quill
--customers Number of customers to generate 100
--vendors Number of vendors to generate 100
--materials Number of materials to generate 100
--users Number of concurrent worker threads 20
--start-year Start year for simulated operations 2025
--years Number of years of activity 1
--operations Total number of operations to execute 15000
--name Custom name for the benchmark run (auto‑gen)

Example

java -jar target/scala-3.6.4/bench-pg-scala-assembly-1.0.0.jar --lib slick --connections 40 --users 20 --operations 15000
java -jar target/scala-3.6.4/bench-pg-scala-assembly-1.0.0.jar --lib zio-quill --connections 40 --users 20

Workload Description

Both benchmarks follow an identical procedure to ensure fair comparison:

  1. Schema initialization – Database and schema should be created before running benchmark.

  2. Data generation – Dimension tables are populated with the specified number of rows using random but realistic data.

  3. Execution – A fixed mix of business transactions (e.g., order placement, invoice generation, stock movements) is executed concurrently across the connection pool. Each transaction consists of multiple SQL statements and touches both dimension and fact tables.

Benchmark Results

To understand how each database library scales with the number of concurrent workers (users), we ran a dedicated experiment.
We fixed all parameters except the number of simulated users (--users), varying it from 5 to 30 in steps of 5.
The workload and environment were kept constant:

Parameter Value
Connections 40
Customers 100
Vendors 100
Materials 100
Start year 2025
Years 1
Operations 5,000

The test was executed on a MacBook Pro (Apple M2 Max 12 vCPUs, 96 GB RAM, SSD storage).

Lower total_duration is better.

Benchmark Chart

The tables below show the complete set of metrics for each library.
All latency values are in seconds (s).

Code Size

One of the project goals is to keep the benchmark harness simple and auditable, while the library‑specific implementations illustrate how different database drivers and ORMs express the same workload. The table below shows the lines of code (excluding blank lines and comments) for each component.

Component Lines of code
Scala benchmark harness 166
Java benchmark harness 237
Go benchmark harness 291
Rust benchmark harness 295
Scala zio-quill implementation 306
Scala slick implementation 330
Java raw-jdbc implementation 486
Java spring-jdbc implementation 552
Rust sqlx implementation 695
Java spring-jpa implementation 707
Rust tokio-postgres implementation 756
Go GORM implementation 799
Go sqlx implementation 872
Rust diesel implementation 880

License

Copyright 2025 bench-pg contributors

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Contributing

Contributions are welcome! If you'd like to add support for another database library, improve the workload realism, or fix bugs, please open an issue or submit a pull request. When adding a new library, ensure that the workload logic remains identical to the existing implementations so that results remain comparable.

Happy benchmarking!