feat(matrix): benchmark multiple wallarm builds side-by-side#2
Merged
Conversation
…ARM_IMAGE CSV Passing WALLARM_IMAGE as a comma-separated list (e.g. 'wallarm:branch-main,wallarm:branch-other') expands the wallarm gateway column into one wallarm@<variant> column per image, so two or more builds can race through the same canonical matrix in a single sweep. Single-value WALLARM_IMAGE is unchanged — column stays plain 'wallarm'. The variant label is derived from the image tag (chars after the last ':'), sanitised for docker, with -N disambiguation on duplicates. The runner injects WALLARM_IMAGE per cell and forces a docker-safe container_name / compose project when the gateway carries an @suffix. load-gateway.sh and aws-clean-cell.sh strip the suffix before resolving gateways/<base>/docker-compose.yaml + setup.sh; per-cell artefacts land at reports/<run-id>/raw/wallarm@<variant>/... Report ordering treats wallarm@* as the wallarm canonical rank with alphabetical variant tiebreak, so the columns sit next to each other. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- scripts/load-gateway.sh: GATEWAY_VARIANT was assigned but never read,
tripping shellcheck SC2034. We resolve the variant suffix lazily via
${GATEWAY%@*}; the assignment was speculative future-use noise.
- gateways/wallarm/README.md: the link to .notes/PROGRESS.md § Iteration 23
pointed at an internal-only path that was already broken on main but
surfaced now that markdown-link-check re-scanned the file. Replace the
link with a plain-text reference.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WALLARM_IMAGEas a comma-separated list (e.g.wallarm:branch-main,wallarm:branch-other) now expands the wallarm gateway column into onewallarm@<variant>column per image — two or more builds race through the same canonical matrix in a single sweep.WALLARM_IMAGEis unchanged; column stayswallarm, no breaking change to existing pipelines.:), sanitised for docker, with-Ndisambiguation on duplicates.How it works
orchestrator/internal/matrixlearnsWallarmVariant,ParseWallarmImageEnv,ExpandWallarmVariants,WallarmImageFor, plusCell.GatewayBase()/Cell.GatewayVariant()helpers.bench runreadsWALLARM_IMAGEfrom the env and threads variants into both--matrix selectionand--matrix canonicalexpansion.runner.RunnerinjectsWALLARM_IMAGE=<cell image>per cell and, when the gateway carries an@variantsuffix, forces a docker-safeBENCH_CONTAINER_PREFIX/BENCH_COMPOSE_PROJECT(@is illegal incontainer_name).scripts/load-gateway.shandscripts/aws-clean-cell.shstrip the@variantto resolvegateways/<base>/docker-compose.yaml, the per-policysetup.sh,FEATURE-MISSING*, and.env. Per-cell artefacts still land atreports/<run-id>/raw/wallarm@<variant>/....wallarm@*as the wallarm canonical rank with alphabetical tiebreak so variant columns sit next to each other; manifest's image probe resolves against the base directory.Example
```bash
WALLARM_IMAGE='wallarm:branch-main,wallarm:branch-other' \
orchestrator/bin/bench run --gateways wallarm,nginx --matrix canonical
columns: nginx, wallarm@branch-main, wallarm@branch-other
```
Test plan
🤖 Generated with Claude Code