gce-load-testing

Repeatable performance-test rigs for rqlite on GCP. One command provisions an ephemeral, fixed-shape cluster plus load generator, with metrics flowing through the Google-Built OpenTelemetry Collector into Cloud Monitoring / Managed Prometheus; one command destroys it.

rqlite nodes (COS VMs) --OTLP push--> GBOC collector VM --> Managed Prometheus
     ^                                                      (persistent
     |  load                                                 dashboards,
loadgen VM                                                   PromQL)

Design principles:

• Fixed shapes, ephemeral rigs. Same machine types, disk, and zone every run; every resource is created per rig and destroyed after. Run-to-run comparability is the entire point.
• DB nodes run rqlite and nothing else. Host networking, no sidecars, no agents. Telemetry is pushed to the collector; OS-level metrics (CPU, disk IOPS) come free from the GCE hypervisor via Cloud Monitoring.
• Everything is stamped from committed config. Cloud-init templates, a pinned collector version, and a recorded rig manifest (runs/<rig>/rig.json) including the rqlite build actually tested — because dev is a moving tag.
• Every metric carries a run_id label (= the rig ID), so one persistent dashboard serves every ephemeral rig: filter by run_id.

Prerequisites

• gcloud authenticated, with a default project and zone configured (or set in config.env — see config.env.example).
• Project owner or equivalent for the one-time setup.sh.
• python3 locally (used to parse JSON responses).

Quickstart

./setup.sh                  # once per project: APIs, VPC, firewall, SA, bucket
./up.sh --nodes 1           # ~3-4 min: single-node rig from rqlite/rqlite:dev
./smoke.sh                  # cluster checks + end-to-end metrics pipeline check
./down.sh                   # tear down. VMs bill until you run this.

A 3-node cluster from a specific image:

./up.sh --nodes 3 --rqlite-image rqlite/rqlite:8.36.3

./status.sh lists every rig currently alive (and billing). ./down.sh --all nukes everything labeled rqlite-perf in the zone.

Rigs record their project and zone in runs/<rig>/rig.json, and down.sh / smoke.sh target that recorded location — so a rig brought up with --project foo tears down correctly regardless of your gcloud default.

Extra rqlited flags (including OTel export)

--rqlite-flags (or RQLITE_EXTRA_FLAGS in config.env) appends arbitrary flags to every node's rqlited command line — OTLP export config, experimental flags, whatever the run needs. The token {collector} expands to the collector VM's internal IP:

./up.sh --nodes 3 --rqlite-flags "-otel-endpoint {collector}:4317 -some-other-flag"

The flags land in runs/<rig>/rig.json, so every run records what it ran with. Without a {collector} reference rigs come up without rqlite telemetry, but smoke.sh still validates the pipeline with synthetic metrics (telemetrygen), which cleanly separates "rqlite isn't pushing" from "the pipeline is broken."

Export is plaintext gRPC inside the VPC (firewalled to rig VMs only).

Running load

./run-load.sh                     # all scenarios in scenarios/, against the latest rig
./run-load.sh write-batched       # just one
./run-load.sh --upload            # also copy results to the GCS bucket

The load tool is rqbench, shipped inside the rqlite image itself — the loadgen drives load with the same image the nodes run. Scenarios are small sourceable files in scenarios/ (statement, request count, batch size, queued/txn flags, concurrent CLIENTS); add a scenario by adding a file. Each scenario produces runs/<rig>/load-<scenario>.log (raw output) and results-<scenario>.json (parameters, start/end timestamps for correlating with Cloud Monitoring, wall-clock statements/sec).

Current limitation: rqbench has no machine-readable output, so throughput is wall-clock-derived and there are no client-side latency percentiles. A -json output mode in rqbench would fix both — the runner has a marked slot for it.

Looking at a run

• Live: Metrics Explorer, PromQL {run_id="<rig-id>"}. GCE hypervisor metrics (CPU, disk, network) appear under the VM resources automatically.
• SSH (all VMs, IAP only): gcloud compute ssh <vm> --tunnel-through-iap. Container logs: sudo docker logs rqlite / sudo docker logs otelcol.
• What was tested: runs/<rig>/rig.json and status-node-1.json (image, reported build version, shapes, flags).

Performance-affecting knobs (keep constant across compared runs)

Knob	Default	Notes
Node machine type	c3-standard-4	--machine-type
Loadgen machine type	c3-standard-8	dedicated VM; never share with nodes
Node disk	pd-ssd, 200 GB	PD IOPS/throughput scale with size — disk size is a performance parameter, not capacity. 200 GB pd-ssd ≈ 6000 IOPS / 96 MB/s write, and rqlite's write path is fsync-bound.
Zone	gcloud default	cross-zone rigs are a future scenario dimension
GBOC version	0.155.0	pinned; bump deliberately in lib/common.sh
--spot	off	same hardware/perf, ~60-90% cheaper, rare preemption. Fine for iteration; use on-demand for baseline runs.

Rough cost: a 3-node c3 rig + loadgen + collector is ~$1.50/hour on-demand. The collector VM (e2-small) is pennies. Custom-metric ingestion for an hour-long run is a few MiB — effectively free.

Cluster formation

Internal IPs are reserved before boot and templated into every node's cloud-init, so all nodes get identical -bootstrap-expect N -join ip1:4002,ip2:4002,... (Raft addresses, rqlite 8.x semantics) and boot order doesn't matter. Readiness is gated on each node's /readyz and the collector's health endpoint (:13133).

Security posture

VMs carry ephemeral external IPs (for Docker Hub / ghcr pulls) but no inbound port is open to the internet: SSH is IAP-range only; rqlite (4001/4002) and OTLP (4317/4318/13133) accept traffic only from other rig VMs, by network tag. Rig VMs run as a dedicated service account whose only powers are writing metrics and logs.

There are two firewall layers: the VPC rules above, and COS's host firewall, which accepts only inbound SSH by default. Because the containers use --network=host, Docker does not open host ports for them, so the cloud-init runcmd adds the iptables ACCEPT rules for rqlite/OTLP ports. A port must be open at both layers to be reachable from another VM.

Layout

setup.sh              one-time project setup (idempotent)
up.sh                 provision a rig            down.sh   destroy a rig
smoke.sh              validate a rig             status.sh what's running/billing
run-load.sh           run load scenarios from the loadgen VM
scenarios/            load scenarios (sourceable .env files)
lib/common.sh         defaults + helpers
templates/            cloud-init for rqlite nodes and the GBOC collector
runs/<rig-id>/        rendered configs + rig.json + /status snapshot (gitignored)
config.env(.example)  local overrides

Roadmap

1. Runner + scenarios — done (run-load.sh + scenarios/). Still open: client-side latency percentiles (needs a rqbench -json mode) and a distinct run_id per scenario.
2. Harvester — merge loadgen client results with Monitoring API aggregates into a summary JSON per run, uploaded to gs://<project>-rqlite-perf.
3. Compare tool — diff a run against a designated baseline; nonzero exit on regression beyond thresholds.
4. Persistent dashboard as code — gcloud monitoring dashboards create from committed JSON, filtered by run_id.
5. Safety-net cleanup — scheduled job deleting any rqlite-perf resource older than N hours; nightly CI trigger.

Troubleshooting

• Service healthy on its own VM but unreachable from other VMs (readiness probes time out while curl localhost:... works on the box): the COS host firewall is blocking it — see Security posture. Check with sudo iptables -L INPUT -n; the cloud-init runcmd should have added ACCEPT rules for the service's ports.
• Collector unhealthy / config rejected: sudo docker logs otelcol on the collector VM. If the attributes processor is unavailable in a GBOC release, stamp run_id via the resource processor plus the googlemanagedprometheus exporter's metric.resource_filters instead.
• c3 unavailable / quota exceeded in your zone: pass --machine-type n2-standard-4 (and keep using it consistently), or request C3 quota.
• First gcloud compute ssh is slow: key propagation takes ~30-60s; scripts retry automatically.
• gcloud compute ssh prompts about OS Login/keys: run it once manually, then rerun the script.