feat(benchmark): add k6 benchmarks for JSONB/encrypted query performance #352
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Use official grafana/xk6-sql with postgres driver instead of unmaintained xk6-pgxpool - Update Dockerfile to Go 1.23 with GOTOOLCHAIN=auto for newer xk6 compatibility - Migrate all benchmark scripts to xk6-sql API (sql.open/db.exec/db.close) - Default to host.docker.internal for macOS Docker compatibility - Add network detection to use --network=host on Linux CI - Create k6_run.sh task with proper USAGE flags for mise - Update CI workflow to set K6_DB_HOST for Linux runners
On Linux, default to 127.0.0.1 (host.docker.internal doesn't resolve). On macOS, default to host.docker.internal. Removes explicit K6_DB_HOST from CI workflow since script now handles it.
tobyhede
changed the title
k6 benchmarks were using the test schema's encrypted table which lacks the required encryption config. Now uses benchmark_encrypted table with proper ste_vec encryption for JSONB. - Add encrypted_jsonb column to benchmark_encrypted table - Add ste_vec search config for JSONB encryption - Update jsonb-insert, jsonb-containment, large-payload scripts
Rename k6 benchmark scripts to clarify they test JSONB with STE-VEC encryption index (as distinct from plain JSONB): - jsonb-insert.js → jsonb-ste-vec-insert.js - jsonb-containment.js → jsonb-ste-vec-containment.js - large-payload.js → jsonb-ste-vec-large-payload.js Update k6:benchmark:continuous task to reference the renamed script.
…benchmarks Separate STE-VEC and non-STE-VEC JSONB benchmarks: - Add encrypted_jsonb_with_ste_vec column with STE-VEC search config - Update jsonb-ste-vec-* scripts to use the new STE-VEC column - Add jsonb-large-payload.js for basic JSONB without STE-VEC index - Update k6_run.sh script choices for new benchmark names This enables comparing encryption overhead with and without searchable encryption indexes.
The randomId() function was generating values up to Number.MAX_SAFE_INTEGER (~9 quadrillion), exceeding PostgreSQL's int4 max of 2.1 billion. This caused "value is out of range for type integer" errors in k6 benchmarks.
Run both jsonb-ste-vec-insert and jsonb-large-payload benchmarks in CI, merging outputs with jq for benchmark-action tracking.
tobyhede
force-pushed
the
ks-benchmarks
branch
from
e81a59d to
7212836
Compare
…puts Separate benchmark-action tracking for proper regression detection: - Throughput (customBiggerIsBetter): rate in iter/s - Latency (customSmallerIsBetter): p95 and p99 in ms Adds p99 metric and configures summaryTrendStats for k6.
tobyhede
force-pushed
the
ks-benchmarks
branch
from
7212836 to
2aeda22
Compare
- Setup EQL and benchmark schema before k6 runs - Start proxy and wait for it to be ready - Add missing pgbench benchmark run step - Move pgbench data to docs/pgbench for consistency
The CI workflow was calling mise tasks like test:wait_for_postgres_to_quack outside the tests/benchmark directory where they are defined, causing a panic in the task parser. Move all setup steps (postgres:setup, benchmark:setup, proxy:up, and wait_for_postgres_to_quack) into the k6:benchmark:continuous mise task which runs with working-directory: tests/benchmark.
Replace encrypted_jsonb column with two new columns sized to match real customer workloads: - encrypted_jsonb_extract (~288KB): structured credit report with 130 tradelines, 45 inquiries, credit scores - encrypted_jsonb_full (~515KB): raw bureau data with 330 consumer records, 750 processing records Add K6_PAYLOAD_MODE env var to test extract, full, or dual (default) column inserts. Dual mode replicates the customer scenario that caused 25s+ timeouts.
- Pre-create results/k6 directory before Docker runs - Run k6 container with --user flag to preserve file ownership - Ensure results directory exists before jq merge commands Fixes "Permission denied" error when writing k6-throughput.json
Replace references to non-existent scripts (jsonb-insert, jsonb-containment) with actual script names (jsonb-ste-vec-insert, jsonb-large-payload).
Remove jsonb-ste-vec-insert from k6:benchmark:continuous task. Simplify output handling to copy single benchmark results.
The benchmark-action needs pull-requests:write to comment on PRs when performance regressions are detected.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
extension
Dockerfile
text-equality, jsonb-containment, large-payload
k6:benchmark)
tracking
Test Plan:
--target=proxy - runs benchmark (16.14 iter/s
locally)
k6:benchmark:continuous after existi