Fix failed provisions - #1054
Merged
Merged
Conversation
- eventql/hyrise: `.dockerignore` for hits.* — the overlayfs cwd exposes
the shared read-only datasets, so `docker build .` was streaming
~75 GB per hits.{json,tsv,csv} into /var/lib/docker as build context
and hitting "write /hits.json: no space left on device" before the
first Dockerfile step ran.
- pinot: bump to 1.5.1; Apache retired 1.5.0 from downloads.apache.org.
- pg_deltax: soft-fail the chmod on hits_*.parquet. The source files
are symlinks into a read-only dataset mount, so `chmod --dereference`
aborts with "Read-only file system"; the mount already publishes
them world-readable (0664) so postgres can read them.
- mariadb-duckdb: skip `chmod o+x /home/ubuntu` when that directory
doesn't exist. Under `set -eu` the missing dir aborted install.
- presto-partitioned: `docker rm -f presto` before `docker run` — the
existence guard occasionally misses a container that dockerd has
re-registered from disk but not yet published to the ps table.
- pg_clickhouse: add `liblz4-dev` and stop swallowing `pgxn install`
failures with `|| true`. The build was failing on `lz4.h` and the
extension never made it onto the system.
- byconity: readiness probe now runs a lightweight CREATE/DROP
DATABASE so `check` waits until tso-server has actually finished
its FoundationDB handshake. Cold provision was failing with
"Can't get process TSO request" even though SELECT 1 succeeded.
- cedardb / cedardb-parquet: export PGHOST=/tmp in check/start/load/
query. The system-scoped systemd unit installer binds the socket
only at /tmp/.s.PGSQL.5432 + 127.0.0.1:5432, but the scripts used
bare `psql` which falls through to postgres' /var/run/postgresql
default and hit "No such file or directory".
- mssql: cap MSSQL_MEMORY_LIMIT_MB to 75 % of guest MemTotal (min 8 GiB). Previous unconstrained SQL Server + bcp columnstore build was OOM-killed at ~19 min (load rc=137). - parseable: drop ingest concurrency 6→3 and chunk size 2500→1000; add curl --retry 5 to survive transient stalls. Prior settings saturated parseable's incoming-batch queue on 16 GiB, then every chunk returned HTTP 408. - druid: maxNumConcurrentSubTasks 10→4 and extend queryable-wait 4 h→6 h. Fewer concurrent index-merge JVMs leaves headroom for the historical to actually load segments after indexing succeeds.
pg_clickhouse: also add libzstd-dev - mariadb-duckdb: `ln` returned EXDEV because the playground exposes hits.parquet via an overlayfs on top of the read-only datasets mount, and /var/lib/mysql lives on the rootfs (separate fs). Try hardlink first (matches original intent — no 14 GB copy), fall back to symlink when EXDEV. - pg_clickhouse: the extension source also #includes <zstd.h> after clearing the lz4.h barrier, so pull libzstd-dev in the same apt install as liblz4-dev.
- cedardb/cedardb-parquet: also export PGUSER=postgres. cedardb's
installer only creates the `postgres` role, and psql's default of
$USER (=root under the playground agent) trips
FATAL: role "root" does not exist
- mariadb-duckdb: scale duckdb-memory-limit to ~50 % MemTotal. Fixed
24 G worked on 32 GB benchmark hosts but OOM-crashed mariadb-server
mid-load on the 16 GiB playground VM (client saw
ERROR 2026 (HY000): TLS/SSL error: unexpected eof while reading).
- pg_deltax: scale load-time work_mem to MemTotal/32 (128 MB floor,
1 GB ceiling). Fixed 1 GB × 4 parallel workers + shared_buffers
crashed the pg_deltax compression pipeline after ~5 min of load
("server closed the connection unexpectedly").
pg_deltax: tighten memory + parallel-worker caps for 16 GiB VMs
- hyrise: aggregate_hash.cpp under -O3 peaks at ~4 GB per cc1plus, so
the default ninja -j$(nproc) SIGKILLs on the 16 GiB / 4-vCPU
playground VM ("Killed signal terminated program cc1plus"). Cap at
-j2 unless MemTotal ≥ 24 GiB.
- pg_deltax: on hosts under 24 GiB, add shared_buffers=1GB,
max_parallel_workers=2, max_worker_processes=8, and drop the
load-time work_mem cap to MemTotal/64 (was MemTotal/32). Fewer +
smaller pg_deltax decoders keep the direct-backfill pipeline from
crossing the physical-RAM ceiling ("server closed the connection
unexpectedly").
pg_deltax.parallel_workers=0 (default) resolves to num_cpus capped at
16 (see src/lib.rs get_parallel_workers, src/copy.rs
handle_copy_from_parquet_parallel). On a 4-vCPU / 16 GiB VM that
splits as 4 decoders + 1 compressor; each decoder holds ~250 MB of
blob state until flush, so the 5-worker × ~250 MB peak plus
shared_buffers + kernel crosses the physical-RAM ceiling and the
backend gets OOM-killed after 4-6 partition flushes ("server closed
the connection unexpectedly").
Pin `pg_deltax.parallel_workers=2` on hosts under 24 GiB so decode
splits as 2 + 1 = 3 workers ≈ 750 MB peak.
hyrise: shrink CSV chunk size for 16 GiB VM
- vm_manager._boot: on _configure_boot failure, tear down the TAP
after killing the fc process so the retry's ensure_tap recreates a
fresh one. Fixes the class of "Open tap device failed: … Resource
busy (os error 16)" retries that got stuck earlier today when
parallel queries triggered simultaneous restores.
- config + monitor: new PLAYGROUND_MAX_READY_VMS cap (default 40),
enforced by the monitor via kick("max-ready-cap") on the
oldest-idle ready VM. Bounds total guest anon memory during
bursts — without it, a mass /api/query stack against N systems
pins N VMs at "ready" until idle_kick_after_sec (10 min), which on
100+ systems piled up hundreds of GB of guest RSS + page cache.
- hyrise/load: scale CSV split size to guest RAM. Fixed 5M rows/piece
worked on 32 GB benchmark hosts but SIGKILL'd hyriseConsole on the
16 GiB playground VM (load rc=137 after 156 s). Drop to 2M rows on
hosts under 24 GiB — smaller per-piece unencoded materialisation
keeps the peak within budget.
trino-partitioned: copy parquet bytes instead of hardlinking symlinks
- hive: BENCH_DURABLE=no covered ClickBench's cold-cycle re-load, but
the playground's pre-snapshot `./stop; ./start` sequence runs
`docker rm -f hive; docker run …` — wiping the embedded Derby
metastore ./load just populated. The snapshot then captures an
empty catalog and every restored /query returns
Failed to open new session: Database clickbench does not exist.
Set PLAYGROUND_SKIP_RESTART_BEFORE_SNAPSHOT=yes so the running HS2
(with the loaded catalog) is what gets snapshotted.
- impala: same class of bug — pre-snapshot `docker compose down; up`
drops catalogd's in-memory catalog. With hms_event_polling_interval_s=0
(needed to sidestep the HMS notification-log RPC bug), catalogd
never re-syncs from HMS on start, so `use clickbench` from every
restored /query fails. Same PLAYGROUND_SKIP_RESTART_BEFORE_SNAPSHOT
gate.
- trino-partitioned: `ln -f` preserved the symlink (GNU default -P),
so trino ≥ 483's local FS provider hit
Malformed Parquet file. Metadata index: -X out of range
[local:///hits/hits_N.parquet]
when reading through the symlink chain. Copy the file bytes instead
for the playground path, keep hardlink for bare-metal runs where
the source is a real file. One-time per-provision cost, reliable
afterwards.
- hyrise: 2M-row chunks still SIGKILL at chunk 10 (~20M rows encoded) because hyrise's per-segment overhead is heavy on the 16 GiB VM. Try 1M-row chunks — 100 pieces at 1M each caps the transient unencoded materialisation at ~1 GB, giving more headroom for the growing encoded catalog. If this still OOMs, hyrise is genuinely outside the playground envelope (same class as umbra). - quickwit/check: gate on an actual /api/v1/hits/search request, not just /api/v1/version. The REST server responds long before the searcher registers with the (single-node) gossip cluster, so the current check was letting the snapshot fire while the cluster still reported "no available searcher nodes" — every restored /query then bounced off that error even with PLAYGROUND_RESTART_AFTER_RESTORE_SNAPSHOT=yes.
The previous version failed the initial post-start check because the `hits` index doesn't exist yet at that point (load creates it). 404 from /hits/search only means the index is missing, not that the searcher is unregistered — the actual failure signal we're trying to catch (500 "no available searcher nodes in the cluster") still fails the gate.
alexey-milovidov
requested a deployment
to
benchmark-approval
July 23, 2026 21:20 — with 
GitHub Actions
Waiting
GitHub Actions
Waiting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.