feat(push-to-registries): add registry layer cache, skip QEMU when host-native

[marians]

Why

Investigating a consistently ~5m30s image build in giantswarm/backstage (measured 5m31s / 5m32s / 5m34s across three PRs — 3s of variance is the tell) turned up two costs on this job's path. Notably, neither is multi-arch emulation: that repo pins platforms: linux/amd64 on both its branch and release jobs, so nothing was ever emulated.

1. There is no layer cache, and there was no way to add one. Each run gets a fresh setup_remote_docker VM and creates a fresh docker-container builder, so the whole Dockerfile is re-executed from scratch every build. A consumer's own RUN --mount=type=cache mounts can't help — they live in the builder state that dies with the VM, so they only ever sped up local rebuilds. For backstage that means re-doing apt-get install build-essential, pip install mkdocs-techdocs-core, and a full yarn workspaces focus --all --production (with node-gyp native compiles) on every single build.

2. QEMU/binfmt was registered unconditionally. The privileged tonistiigi/binfmt --install all pull ran even for single-platform builds, registering handlers that can never be used — no RUN step is emulated when every target platform matches the build host.

What

cache (new, defaults to off)

cache: registry adds --cache-from / --cache-to type=registry, persisting BuildKit's layer cache as an OCI artifact at <first-eligible-registry>/<image>:buildcache<tag-suffix>. cache-ref overrides the ref (e.g. to scope per branch); cache-mode (default max) sets the exporter mode — max matters, since min would skip exactly the intermediate RUN layers that are expensive.

Defaults to off, so no existing consumer changes behaviour.

Two exporter attributes worth calling out:

• image-manifest=true,oci-mediatypes=true — ACR (gsoci/gsociprivate) rejects the exporter's default manifest-list format, so the cache has to go up as a plain OCI image manifest.
• ignore-error=true — a failed cache write must not fail a build whose image is already pushed, and must not trigger the surrounding four-attempt retry loop.

Requires push: true, since the export reuses the credentials the push path already set up; with push: false the parameter is ignored and the build says so.

QEMU gating

binfmt registration now happens only when at least one target platform differs from docker version --format '{{.Server.Os}}/{{.Server.Arch}}'. Multi-arch builds are unchanged. If the host platform can't be determined, handlers are registered as before — the failure mode stays "registered unnecessarily", never "cross build without emulation".

Caveats (documented in docs/job/push-to-registries.md)

• Cache mounts are not exported, only layers. A layer hit skips the RUN entirely so its mount is irrelevant; a miss re-runs it against a cold mount, so a lockfile bump still costs full price.
• The default cache ref is shared and mutable: concurrent builds race on the tag (last write wins, costing only a later miss), and any build with push access can write it. cache-ref isolates if a repo needs that.
• The :buildcache tag shows up as an extra tag in the repository listing.

Testing

circleci orb pack + circleci orb validate pass. The emulation-detection logic was unit-checked against linux/amd64-on-amd64 (skip), multi-arch (register), linux/arm64-on-amd64 (register), linux/arm64/v8-on-arm64 (registers — a harmless false positive, erring safe), and unknown host (register).

Draft until validated end-to-end against giantswarm/backstage via dev:buildkit-registry-cache. I'll post the before/after job timings here.

---

Measured on giantswarm/backstage (#2012)

Validated end-to-end via dev:buildkit-registry-cache. push-to-registries duration:

run	duration
baseline (main, orb 9.6.0, no cache)	~5m32s
cold (cache written, QEMU skipped)	5m15s
warm (cache hit)	2m03s
re-run after base-image republish	5m40s (cold — confounded, see below)
warm, current revision, stable base	2m08s

3m29s faster — 2.7×.

Notable: the cold run came in 17s under baseline even while uploading 554 MB of cache. That delta is the QEMU gating — the tonistiigi/binfmt pull it now skips cost roughly as much as the cache export added.

Verification, not just timings:

• The cache landed at gsoci.azurecr.io/giantswarm/backstage:buildcache as application/vnd.oci.image.manifest.v1+json with a application/vnd.buildkit.cacheconfig.v0 config, 24 blobs / 554 MB. This confirms image-manifest=true is genuinely required — ACR would have rejected the exporter's default format, and ignore-error=true would have hidden that as a silent no-op.
• The warm run pushed a real image (0.178.1-dev.test-image-build-cache.2026-08-03.11-53-40.hc44a292), so the speedup is a cache hit, not a skipped push.
• The warm run's commit changed backend source, so bundle.tar.gz differed while skeleton.tar.gz stayed byte-identical — i.e. the realistic PR case, not an empty re-run.

---

Review follow-ups (see review reply)

Fixed since the review: cache ref no longer varies by build type (was a real fork bug) and never resolves to the China mirror; a swallowed cache-export failure now emits a loud warning; cache-ref is validated. Docs gained the untagged-manifest retention prerequisite, a proper trust-model section, and a branch-jobs-only recommendation so signed release artifacts are never assembled from a shared mutable cache.

Additional caveat found while validating: a floating base tag defeats the cache entirely. node:24-trixie-slim was republished mid-validation, making two builds of an unchanged yarn.lock both fully cold. Treat the speedup as "fast between base-image republishes" and pin the base by digest for consistent hits.

Bound on orphan growth: a fully warm build re-exports byte-identical content, leaving the manifest digest unchanged, so untagged orphans accrue per content change (base bumps, lockfile changes) rather than per build.

Still open: registry egress is unmeasured — it needs the buildx byte counters from the job log.

[piontec]

Reviewed with a focus on artifacts the cache leaves behind and on total cost. The QEMU gating is a clean win — platforms is resolved well before the gate, {{.Server.Arch}} is the right field (GOARCH amd64, not docker info's x86_64), and it fails safe on unknown host and on variant mismatch. No notes there.

Three things on the cache side.

1. Untagged cache manifests accumulate with no cleanup. Each export PUTs a new manifest and moves the :buildcache tag; the previous manifest becomes untagged, not deleted. ACR does not GC untagged manifests by default (retention is Premium and opt-in per registry). Referenced blobs dedupe, so growth is the per-build delta — but for backstage the layers that change are exactly the big ones. Every branch build leaves a permanent orphan, multiplied by opted-in repos × builds/day. Also permanent: the :buildcache tag itself, if a repo ever stops using cache: registry or renames the image.

Suggest making untagged-manifest retention on gsoci/gsociprivate an explicit prerequisite in the docs (az acr config retention update --type UntaggedManifests --days 7 --status enabled, or a purge task), rather than leaving it implied.

Related: the concurrency note in the docs undersells the race. It isn't only "a later cache miss" — a build with cache-mode: min, or a branch that dropped a stage, replaces the rich cache wholesale and orphans the loser's blobs.

2. The derived cache ref isn't stable, so it forks silently. head -n1 /tmp/.eligible_registries reads a list written after both the visibility filter and the push_dev filter. If the first visibility-eligible registry has push_dev=false, dev builds resolve to registry #2 and release builds to registry #1 — two forever artifacts per image, and release builds never reuse the cache the far more frequent branch builds paid to write. Same fork on a visibility flip or a reorder of REGISTRIES_DATA_BASE64: old tag orphaned, builds silently cold, no error.

Deriving from the unfiltered registries list, or hard-preferring gsoci/gsociprivate, would make it byte-stable across build types — which it has to be to be worth anything.

3. The cost is measured on one side only. 3m29s/build is convincing. Missing: ~554 MB of standing ACR storage per opted-in image, orphan growth per build, and ACR egress on every warm build — the cache blobs now come down from ACR to be pushed to the other registries, where before they were produced locally. Probably still net-positive against CircleCI credits, but worth the estimate before recommending this beyond backstage.

Two smaller ones:

• ignore-error=true is the right call per build, but if ACR ever starts rejecting the export (policy, quota, auth scope) every build reverts to full cost forever with no signal. Grepping the buildx output for a cache-export failure and emitting a loud non-fatal warning would cover it.
• cache_ref="<<parameters.cache-ref>>" is interpolated unvalidated. Low severity given config.yml is already trusted, but the job validates platforms with a regex and image-login-to-registries validates env-var names, so it's inconsistent with local practice.

One trust-model note: image-login-to-registries logs into every registry with shared context credentials for every repo using this orb, so any CI job in the org can write any image's :buildcache, which --cache-from then imports into release builds that get cosign-signed and carry --attest type=provenance. The capability isn't new — those credentials can already push image tags — but the stealth is: content changes without any tag anyone inspects moving, and the provenance attestation becomes a false statement. "any build with push access can write it" in the docs reads as a nuisance; worth saying the layer cache sits inside the trust boundary of every repo sharing the push credentials.

Nit: docs say "Since v9.7" in three places while the CHANGELOG entry is under [Unreleased].

[marians]

Thanks — this caught a real bug and changed the recommended usage. All six points addressed; two produced new data, one I can only partly answer.

2. Unstable cache ref — confirmed bug, fixed

You were right, and it was the important one. Simulated with a registries-data where the first visibility-eligible registry has push_dev=false:

dev=true   old=quay.io           new=gsoci.azurecr.io
dev=false  old=gsoci.azurecr.io  new=gsoci.azurecr.io

Exactly the fork you described. Now derived without the push_dev filter.

I initially got the fix wrong in a way worth recording: my first attempt used the visibility filter alone, which also dropped the split-china exclusion — so for any repo listing Aliyun ahead of gsoci with matching visibility, the cache would have targeted the China mirror, putting back on the critical path exactly the Pacific crossing split-china-push exists to avoid. Latent rather than observed (backstage resolved to gsoci throughout), but your "hard-prefer gsoci/gsociprivate" suggestion was the better answer and is what's implemented now: China mirror excluded unconditionally (not gated on split-china-push, since that parameter also varies by build type and would reintroduce the fork), GS registries preferred by name, first remaining visibility match as fallback, no cache if nothing survives. Verified across four registry layouts.

4. Silent forever-failure — fixed, and it's worse than described

Reproduced locally rather than guessing at a grep pattern, and found an extra failure mode: the exporter retries with escalating backoff (30s, 61s, 93s…) before giving up. So a broken cache registry doesn't only revert every build to full cost — it adds minutes per build while reporting nothing.

Confirmed ignore-error=true swallows it completely (EXIT=0 against a real ACR insufficient_scope rejection). The job now greps the output for the failed export vertex and warns loudly. Marker verified in both directions — present on a rejected export, absent on a successful one — so it can't fire spuriously on every build.

1. Untagged manifest accumulation — documented, and less severe than feared

Retention is now an explicit prerequisite with the az acr config retention update command, plus a note that the :buildcache tag itself is permanent. Concurrency note rewritten; you were right that "only a later cache miss" undersold the cache-mode: min / dropped-stage case.

New data that bounds the growth: a fully warm build re-exports byte-identical content, so the manifest digest doesn't change and no new orphan is created. Two consecutive runs both left sha256:4617b936afb9a06e… in place. So orphans accrue per content change — base-image bumps, lockfile changes — not per build. On a repo with a pinned base and stable dependencies that's a handful per month, not one per build.

3. One-sided cost — partly answered, and the answer got worse

Standing storage is 554 MB per opted-in image. Orphan growth is bounded as above.

Egress I have not measured and won't guess at — it needs the buildx byte counters from the job log, which I don't currently have access to. Documented as a real cost to account for before opting in many repos.

The estimate also has to absorb something I found while validating, which cuts the other way:

Floating base tags silently defeat this

Two builds of an unchanged yarn.lock, three days apart, were both cold — because node:24-trixie-slim was republished at 2026-08-05T08:39:10Z in between. A moved FROM digest invalidates every layer beneath it. -slim tags get Debian security rebuilds often.

So the honest claim is "fast between base-image republishes", not "fast every build", which lowers the amortised benefit relative to the standing storage cost. Docs now recommend pinning by digest with Renovate managing it. This is arguably the single most important caveat for anyone evaluating the feature.

Trust model — promoted, and it changed the design

Your framing was right and the bullet was too weak. It now has its own section stating plainly that the cache sits inside the trust boundary of every repo sharing the push credentials, that content can change without any inspected tag moving, and that the provenance attestation becomes a false statement.

That drove a design change: the recommended setup is now branch/PR jobs only, with release-tag jobs left on cache: off. Branch builds are the frequent ones so they capture nearly all the saving, and signed, provenance-attested release artifacts are then never assembled from a shared mutable cache.

5. Unvalidated cache-ref — fixed

Validated against the OCI reference charset in a dedicated step, matching the existing platforms check. A , would otherwise have silently injected a further exporter attribute.

Nit — fixed

Dropped the version references rather than adding a version heading, since releases are automated. Also corrected an adjacent "Since v8.2" line that my QEMU change had made inaccurate for single-arch builds.

Measurements

run	push-to-registries
baseline (main, orb 9.6.0, no cache)	~5m32s
cold, cache written	5m15s
warm	2m03s
re-run after base-image republish	5m40s (cold — confounded)
warm, this revision, stable base	2m08s

The 5m40s row is the base-image confound, not a regression — I misread it as one at first, on a size comparison against a digest I had never recorded. Corrected above.