Feature/2.0.4 container runtime - #7
Merged
Merged
Conversation
Introduce a backend abstraction so benchmarks can run without Kubernetes. Selected via `backend: kubernetes | container | podman | docker` in config; kubernetes remains the default so existing configs are unaffected. The enabling change is runtime/environment.py, which lifts the container env-var contract out of the Helm template into Python. Previously that contract existed only in Go template syntax, so a non-Kubernetes backend had no way to reuse it and it could not be unit tested. Both backends now derive their environment from the same code. Verified parity by diffing helm-rendered Job env against the Python builder across Oracle and SQL Server, TPC-C and TPC-H, with and without Pure Storage: identical keys and values in every combination. That diff caught a real bug. Helm rendered total_iterations 10000000 as "1e+07", which reaches `diset tpcc total_iterations` in the TCL scripts and is not a valid Tcl integer. Fixed in both chart copies with printf %.0f. This is the same class of YAML scientific-notation issue as 36c9a67. Validated against live hardware: - 8 Oracle 26ai targets in parallel, all completed and parsed, aggregating 173,444 TPM / 84,303 NOPM - 2 SQL Server 2025 targets, both completed and parsed - status, log retrieval, duration and cleanup all correct on both Tests: 138 -> 178. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Route build, run, status, logs, results, report and clean through the selected backend rather than calling helm and kubectl directly. Kubernetes stays the default, and its rendered manifests are byte-identical to before. Adds runtime/resolve.py with backend-agnostic test-id resolution and completion waiting, replacing the Kubernetes-only _wait_for_jobs. The new waiter also prints the database error extracted from a failing workload's log instead of leaving the user to go and read logs themselves. aggregate_results now takes a backend instead of a namespace, which is what lets the results and report commands work on either path. Fixes a phase-label bug found by running the wired CLI end to end: containers were labelled phase=run while every caller filters on phase=load, the name Helm uses. --wait therefore never observed completion and timed out at 400s against containers that had exited successfully in 65s. The container backend now normalises the phase for both labelling and querying, so container names also match the Helm scheme (hdb-load-00-<id>). Verified the full workflow on live Oracle via podman: run --wait, status, results, report (210KB scorecard with correct figures) and clean. Tests: 178 -> 181. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The version was hardcoded in 22 places across entrypoint.sh, both Dockerfiles
and both chart copies, split across the image (where HammerDB installs) and
the chart (where scripts get mounted). A mismatched pair produces a container
that starts and then cannot find its TCL scripts.
Now defined once as ARG HAMMERDB_VERSION in the Dockerfile, exported as
HAMMERDB_HOME, and mirrored by DEFAULT_HAMMERDB_VERSION for chart rendering.
entrypoint.sh reads HAMMERDB_HOME and falls back to a glob probe, so it still
works in images built before that variable existed. A future bump is one
build arg and one constant.
Also fixes two latent coupling issues found on the way:
- entrypoint.sh lived inside the scripts directory that gets overmounted at
runtime; moved to /usr/local/bin
- the Oracle Instant Client check was pinned to instantclient_21_11, so a
client upgrade would have silently failed the check; now version-agnostic
Validated against live hardware with locally-built 6.0 images:
- Oracle 26ai read its existing 5.0-built 10,000-warehouse schema with no
rebuild required, which was the main upgrade risk
- log formats are unchanged, so the existing parsers work untouched
- same-host comparison: 5.0 19,398 TPM vs 6.0 18,882 TPM, a 2.7% delta
consistent with run-to-run variance on a 1-minute test
- SQL Server 2025 also verified end to end
Images are built and tested locally only; publishing to a registry is a
separate step. Chart appVersion moved to 6.0.
Adds test_hammerdb_version.py, including a chart-sync test that fails if the
repo-root chart and the packaged chart drift apart. get_chart_path() prefers
the packaged copy, so that drift would otherwise mean fixes never reach
anyone installing from PyPI.
Tests: 181 -> 193.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by deploying to a real OpenShift 4.19 cluster: every Kubernetes run
failed with "Script '/opt/HammerDB-5.0/scripts/load_test_tprocc.tcl' not
found". The 6.0 work moved the chart's mountPath to /opt/HammerDB-6.0/scripts
while the published image is still 5.0, so the ConfigMap landed the TCL
scripts somewhere the entrypoint never looked.
This is the exact image/chart skew the HAMMERDB_HOME indirection was meant to
prevent, and it slipped through because the container backend reads
HAMMERDB_HOME from the image while the chart hardcodes its own path. They are
set by different artifacts and can always disagree.
Two changes:
- entrypoint.sh now searches for the mounted scripts ($HAMMERDB_HOME/scripts,
then any /opt/HammerDB-*/scripts, then /scripts) instead of requiring an
exact match. HAMMERDB_SCRIPT_DIR overrides. Any chart version now works
against any image version.
- the chart default tracks the published image (5.0), not the version this
repo builds (6.0). Pointing it at a path the published image lacks broke
every out-of-the-box run. PUBLISHED_HAMMERDB_VERSION moves to 6.0 when the
6.0 images are pushed.
Verified on OpenShift 4.19.7 in namespace hammerdb-scale-dev: 2 Oracle targets,
both jobs Complete, results collected through the CLI (27,332 TPM total).
Tests: 193 -> 194.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…onfigs Four usability and security changes, all verified on the OpenShift cluster. Credentials no longer appear in the Job spec. They were plain `value:` entries, so anyone with namespace read access could recover them with `kubectl describe job`. A per-release Secret now holds them and the Job uses secretKeyRef. Confirmed on the live cluster: zero occurrences of the password in the rendered Job. Set kubernetes.use_secrets: false to opt out. The chart now emits a restricted-compliant securityContext. OpenShift's SCC injects an equivalent one, so this changes nothing there, but on plain Kubernetes enforcing the restricted Pod Security Standard the chart would otherwise be rejected at admission. runAsUser is deliberately left unset, because OpenShift assigns a UID from the namespace range and pinning one conflicts with that. validate --from-cluster runs the reachability probe from where the benchmark actually executes, as a pod on Kubernetes or a container locally. The existing check runs from the workstation, which is a different network position: a config could pass validation and then have every job fail on connectivity, failing late and pointing nowhere useful. Minimal configs now work. The oracle:/mssql: block was mandatory whenever a type was set, but every field in it has a working default, so it was ~15 lines restating them. Omitting it now means "use the defaults". A working config is 26 lines rather than ~90; verified by running a real benchmark from one. Writing the tests caught a defect in my own templates: `X | default true` treats an explicit `false` as absent, so both opt-out flags were silently ignored. Fixed by comparing against "false" directly. Verified on OpenShift 4.19.7 with enforce=restricted: jobs deploy, run and complete, results collected through the CLI (29,671 TPM). --from-cluster verified on both backends. Tests: 194 -> 202. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…asking Correctness items from the initial review, several of which affect the tool as shipped today. kubectl version detection was broken. Every call used --short, which kubectl removed in 1.28, so `version` and `validate` silently reported "version check failed" against any current install. Now tries JSON output first and falls back for older builds. validate also stopped assuming Kubernetes: with a container backend it checks the container runtime instead of demanding helm and kubectl, and it checks Secret permissions when credentials use a Secret. The CLI version was wrong. constants.py said 2.0.2 while pyproject said 2.0.3. Reading installed metadata alone was not enough either, since an editable install reports whatever version was current when pip last ran (observed as 2.0.0 against a 2.0.3 tree). Now reads pyproject.toml when in a source checkout, with metadata as the fallback. Reports 2.0.3 correctly. The chart is no longer duplicated. templates/, scripts/, Chart.yaml and values.yaml at the repo root are now symlinks into src/hammerdb_scale/chart/. The copies had already drifted (collect_pure_metrics.py differed), and since get_chart_path() prefers the packaged copy, a fix applied only at the root would never reach anyone installing from PyPI. Verified helm renders identically from both paths and the built wheel still contains all 25 chart files. Error masking: a backend that cannot be queried now says so. `results` used to report "No results found" when the real cause was an unreachable cluster or expired credentials, sending the user to look in entirely the wrong place. Housekeeping: scale configs moved to examples/scale-tests/, 1.2MB of generated HTML reports removed from version control (committed by an earlier `git add -A` of mine), and results/ plus *.html added to .gitignore. Adds test_cli.py: 18 command-level tests with the backend mocked, covering the orchestration layer that had none. That is where the bugs have been, including both recent upstream fixes and two found during this work. Verified end to end on both backends after the changes: podman against Oracle, and OpenShift 4.19 with 2 targets. Tests: 202 -> 222. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PostgreSQL joins Oracle and SQL Server as a first-class database: schema, TCL scripts, parser, entrypoint dispatch, connectivity check, Helm ConfigMap and Secret wiring, and report labelling. The base image could not actually run PostgreSQL despite advertising it. The Dockerfile has claimed "mssql,postgresql,mysql" support since the start, but HammerDB bundles Pgtcl and links the system libpq at runtime, so the driver failed with "Failed to load Pgtcl - libpq.so.5: cannot open shared object file". Adding libpq5 to the base image fixes it; no extension image is needed the way Oracle needs one. Stored procedures default to on, HammerDB's recommended form for PostgreSQL TPROC-C. The setting is exposed because it must match between build and run: the driver calls procedures that only exist if the schema was built with them. Fixes a test-ID ordering bug that affects all backends, found when `results` reported zero metrics after a successful run. find_test_ids relied on the container runtime's ps ordering, which is not guaranteed, so a stale build run surfaced ahead of the run just finished. Now sorted explicitly by creation time, handling podman's unix timestamps and docker's RFC3339 strings. Fleet provisioning: examples/postgres-setup/provision_postgres.sh builds a host from bare RHEL 9 to a tuned PostgreSQL 18.4. Idempotent, and --force rebuilds the cluster so a fleet is provably identical rather than whatever was there before. Sizing derives from the host (shared_buffers 25% of RAM), and it opens 5432 in firewalld, which was blocking access on 7 of the 8 hosts. All 8 PostgreSQL-Bench hosts provisioned identically: PostgreSQL 18.4, 32GB shared_buffers, 32GB max_wal_size, data checksums on, 48 cores / 125GB RAM. Verified end to end: 8 targets in parallel at 100 warehouses, all completed, 2,977,855 TPM / 1,301,691 NOPM aggregate with tight per-host variance confirming uniform configuration. Kubernetes rendering verified for the PostgreSQL path. Tests: 223 -> 234. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Turns two working local images into a repeatable release process. hack/build-images.sh builds the base image, then the Oracle extension FROM that base, verifies both, and only then applies moving tags. Ordering is not incidental: Dockerfile.oracle defaults to BASE_IMAGE=sillidata/hammerdb-scale :latest, so a careless build produces an "Oracle 6.0" image containing the published 5.0 base. The script wires BASE_IMAGE explicitly and then checks that hammerdbcli actually reports v6.0, which catches that whole class of error in two seconds. Verification covers the failures seen during this work: HAMMERDB_HOME resolves, hammerdbcli reports the expected version, libpq is present (without it PostgreSQL fails at runtime with a confusing Pgtcl error), the entrypoint is where the image expects it, and sqlplus works for Oracle. Nothing is tagged :latest until all of it passes. Oracle Instant Client version is now four build args instead of being spelled out in the download URLs, install path, ld.so config and libclntsh soname. That was the same hardcoding problem already fixed for HammerDB: a client upgrade meant editing five lines with no way to confirm you had caught them all. Both images now carry OCI provenance labels, so a published artifact traces back to a git commit. The release script marks a dirty tree as such. Also: renamed dockerfile to Dockerfile (the lowercase name breaks tooling that looks for the standard one), and corrected the database.support label, which claimed PostgreSQL support for as long as the image lacked libpq. Architecture is x86_64 for both, documented rather than implicit. HammerDB 6.0 does ship ARM64 tarballs and the base image would build, but Oracle Instant Client is x86_64-only, so a multi-arch base alongside an x86-only Oracle image would fail confusingly for ARM users. Verified: build-images.sh runs clean, all checks pass, and all three engines (Oracle, SQL Server, PostgreSQL) complete real benchmarks on the release- tagged images. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 6.0 base and Oracle images are now on Docker Hub under sillidata, tagged 6.0, 6, 2.0.3-hdb6.0 and latest, so PUBLISHED_HAMMERDB_VERSION can move off 5.0 and the chart can mount scripts at the matching path. Verified before moving it that an unmodified CLI 2.0.3 install, whose chart hardcodes /opt/HammerDB-5.0/scripts, still runs against a 6.0 image: the entrypoint search resolves HAMMERDB_HOME from the image and SCRIPT_DIR from wherever the scripts were actually mounted. Full TPC-C runs completed on both images against live SQL Server and Oracle, and 2.0.3 ships TCL parse scripts byte-identical to these, so existing users are not broken by latest moving to 6.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.0 reports per-transaction response time distributions from reservoir sampling, so tail percentiles are exact rather than bucket approximations. The TCL already asked for them and TPROCC_TIMEPROFILE already flowed through, but time_profile defaulted to false and the parser discarded the block, so the data reached the container logs and stopped there. Adds parse_transaction_timings, carries the result through the aggregator, and renders a percentile table plus a p50/p95/p99 chart. Sits directly above the storage section: array latency is hard to argue from without the application's view of the same moment, and the max column surfaces stalls that averages hide (one run showed SLEV at 12,639ms max against a 2.96ms median). Cross-target aggregation weights percentiles by call count but keeps max as the worst seen anywhere, since a single stall on one target is the finding rather than noise to be averaged away. time_profile now defaults on. Measured on an 8-VU SQL Server run the overhead sat inside run-to-run variance: 100,371 TPM profiled against 98,575 and 106,322 unprofiled on the same host and schema. Every layer treats missing timings as normal, matching how pure_metrics already behaved: profiling off renders a report with the section absent rather than empty. Verified across all four combinations of timing/array data present. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ports Four defects that produced confidently wrong output rather than an error. validate --from-cluster probed every non-Oracle target on the SQL Server port, so a healthy PostgreSQL fleet was reported unreachable. Port now resolves per database type, preferring the target block, then the shared defaults, then the well-known port. Container runs carried no deployment-name label, so two configs sharing a host saw each other's test IDs and `results` could report another run's numbers. Applies and filters on the same hammerdb.io/deployment-name label the Kubernetes path already uses. `clean --all` is scoped the same way, since "everything" should mean this config's containers, not a colleague's. Runs created before the label fall back to the unscoped query rather than appearing to have vanished. The :z SELinux relabel was applied to every script mount. It is correct on enforcing Linux hosts and has historically been rejected by Docker Desktop's VM, so it made the first run on a Mac fail at mount time. Now conditional on /sys/fs/selinux reporting enforcing. --file also accepts -c and --config; -c is what people reach for by habit. Verified against live podman: two configs run back to back on the same host now each see only their own test ID and report their own throughput. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The container backend removed the Kubernetes prerequisite, but both first-contact surfaces still funnelled every user into needing a cluster. The README listed a Kubernetes cluster under Requirements, and `init` asked for a namespace and pod resource limits without ever asking where the workers should run. A user with no cluster was being interrogated about cluster settings. README now lists Python plus podman or docker as the requirements, with Kubernetes as a section for people who already have one. Adds a platform support table recording that Oracle is x86_64-only, since Oracle publishes no ARM64 Instant Client and Apple Silicon users otherwise discover this at run time. Documents that Oracle needs a locally built image for licensing reasons, and that schema build time dominates the wall clock. `init` asks where to run the workers as its second question, ordered so the runtime actually installed comes first, and emits only the relevant backend block. PostgreSQL was missing from the wizard's database list and, worse, fell through to the mssql branch of the config template, so choosing it produced ODBC settings. Adds container-backend examples for SQL Server and PostgreSQL, and tests that every shipped example still loads, so a stale example fails CI rather than a customer's first run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2.0.3 is already on PyPI, so this branch cannot ship under that version. Bumps the CLI to 2.0.4 and brings the Helm chart with it: the chart was still at 2.0.2, a version behind the last release, so `helm list` reported a version that did not match the tool that deployed it. Adds tests pinning chart, pyproject and CLI versions together, and appVersion to the HammerDB the published images actually carry. CHANGELOG had no entry since 2.0.1 in March. Adds 2.0.2, 2.0.3 and a full 2.0.4 entry covering the container backend, PostgreSQL, HammerDB 6.0 and the response time metrics, with the known limitations stated plainly: Oracle is x86_64-only, Oracle needs a locally built image, the container backend drives from one host, and only Linux with podman has been exercised end to end. Drops "on Kubernetes" from the package description and the remaining Kubernetes-only wording in the README workflow diagram and command table, which contradicted the container-first framing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CONFIGURATION.md, CONTAINER-IMAGES.md, SECURITY.md, and USAGE-GUIDE.md still described the tool as Kubernetes-only and referenced HammerDB 5.0, predating the podman/docker backend and the 6.0 upgrade. Added the missing backend, container, and PostgreSQL field references, corrected stale version numbers, and split backend-specific behavior (validate layers, troubleshooting, resource limits) where the two paths actually differ. Removed docs/oracle-scale-environment.md, a personal handoff note with a live database password checked into public docs. Rewrote em dashes and fragment-heavy bullet lists into plain sentences throughout.
_parse_ts built a colonless UTC offset (-0600) from podman's native Go timestamp format. datetime.fromisoformat() only accepts that form on Python 3.11+; 3.10 requires the colon, so container duration lookups silently returned None on 3.10. Insert the colon when reassembling the offset. Pinned ruff to <0.16.0: 0.16 expands the default rule set well beyond 0.15's, and the unpinned >=0.15.0 constraint let CI resolve a version that failed lint against code never checked against those rules. Ran ruff format under the pinned version to clear the resulting formatting drift across 11 files; no logic changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.