Skip to content

feat(matrix): benchmark multiple wallarm builds side-by-side#2

Merged
denisqsound merged 2 commits into
mainfrom
feat/wallarm-multi-image-matrix
May 19, 2026
Merged

feat(matrix): benchmark multiple wallarm builds side-by-side#2
denisqsound merged 2 commits into
mainfrom
feat/wallarm-multi-image-matrix

Conversation

@w-kerman

Copy link
Copy Markdown
Contributor

Summary

  • Passing WALLARM_IMAGE as a comma-separated list (e.g. wallarm:branch-main,wallarm:branch-other) now expands the wallarm gateway column into one wallarm@<variant> column per image — two or more builds race through the same canonical matrix in a single sweep.
  • Single-value WALLARM_IMAGE is unchanged; column stays wallarm, no breaking change to existing pipelines.
  • Variant label is derived from the image tag (chars after the last :), sanitised for docker, with -N disambiguation on duplicates.

How it works

  • orchestrator/internal/matrix learns WallarmVariant, ParseWallarmImageEnv, ExpandWallarmVariants, WallarmImageFor, plus Cell.GatewayBase() / Cell.GatewayVariant() helpers.
  • bench run reads WALLARM_IMAGE from the env and threads variants into both --matrix selection and --matrix canonical expansion.
  • runner.Runner injects WALLARM_IMAGE=<cell image> per cell and, when the gateway carries an @variant suffix, forces a docker-safe BENCH_CONTAINER_PREFIX / BENCH_COMPOSE_PROJECT (@ is illegal in container_name).
  • scripts/load-gateway.sh and scripts/aws-clean-cell.sh strip the @variant to resolve gateways/<base>/docker-compose.yaml, the per-policy setup.sh, FEATURE-MISSING*, and .env. Per-cell artefacts still land at reports/<run-id>/raw/wallarm@<variant>/....
  • Report ordering treats wallarm@* as the wallarm canonical rank with alphabetical tiebreak so variant columns sit next to each other; manifest's image probe resolves against the base directory.

Example

```bash
WALLARM_IMAGE='wallarm:branch-main,wallarm:branch-other' \
orchestrator/bin/bench run --gateways wallarm,nginx --matrix canonical

columns: nginx, wallarm@branch-main, wallarm@branch-other

```

Test plan

  • `go test ./...` in `orchestrator/` (5 new test cases in `matrix_test.go` covering parse / expand / sort / cell helpers + canonical expansion)
  • `go vet ./...` clean
  • `bash -n` on `scripts/load-gateway.sh` and `scripts/aws-clean-cell.sh`
  • `bench run --dry-run` plans the right cell count with `WALLARM_IMAGE='a,b' --gateways wallarm,nginx --matrix canonical --loads p1-baseline` (39 = 13×3 cells)
  • Single-value `WALLARM_IMAGE` still produces a plain `wallarm` column (no breaking change)
  • End-to-end AWS sweep with two real wallarm images to confirm compose-up, container naming, and report rendering

🤖 Generated with Claude Code

w-kerman and others added 2 commits May 19, 2026 16:19
…ARM_IMAGE CSV

Passing WALLARM_IMAGE as a comma-separated list (e.g.
'wallarm:branch-main,wallarm:branch-other') expands the wallarm gateway
column into one wallarm@<variant> column per image, so two or more
builds can race through the same canonical matrix in a single sweep.
Single-value WALLARM_IMAGE is unchanged — column stays plain 'wallarm'.

The variant label is derived from the image tag (chars after the last
':'), sanitised for docker, with -N disambiguation on duplicates. The
runner injects WALLARM_IMAGE per cell and forces a docker-safe
container_name / compose project when the gateway carries an @suffix.
load-gateway.sh and aws-clean-cell.sh strip the suffix before resolving
gateways/<base>/docker-compose.yaml + setup.sh; per-cell artefacts
land at reports/<run-id>/raw/wallarm@<variant>/...

Report ordering treats wallarm@* as the wallarm canonical rank with
alphabetical variant tiebreak, so the columns sit next to each other.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- scripts/load-gateway.sh: GATEWAY_VARIANT was assigned but never read,
  tripping shellcheck SC2034. We resolve the variant suffix lazily via
  ${GATEWAY%@*}; the assignment was speculative future-use noise.
- gateways/wallarm/README.md: the link to .notes/PROGRESS.md § Iteration 23
  pointed at an internal-only path that was already broken on main but
  surfaced now that markdown-link-check re-scanned the file. Replace the
  link with a plain-text reference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@denisqsound
denisqsound merged commit ab9e195 into main May 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants