Build failure for master branch

[jeesmon]

Docker build is failing for last few commits to master. Can it be fixed please so that we can get latest build with vulnerability fixes?

Pipeline link: https://app.circleci.com/pipelines/github/prometheus/pushgateway/937/workflows/967a5dcb-a3c6-4cf0-a4d4-f5c97934892a/jobs/2542

Error:

 => => extracting sha256:3d9e5d46732a2c40e9c46d378f72940a24d14671c0a76cb2  0.0s
 => [2/4] COPY --chown=nobody:nobody .build/linux-armv7/pushgateway /bin/  0.1s
 => ERROR [3/4] RUN mkdir -p /pushgateway && chown nobody:nobody /pushgat  0.2s
------
 > [3/4] RUN mkdir -p /pushgateway && chown nobody:nobody /pushgateway:
#5 0.137 /bin/sh: mkdir: Function not implemented
------
process "/bin/sh -c mkdir -p /pushgateway && chown nobody:nobody /pushgateway" did not complete successfully: exit code: 127
make: *** [Makefile.common:223: common-docker-armv7] Error 1

[beorn7]

Hmm, weird. I'll look into it at my next convenience. Help from any docker expert welcome in the meantime.

[beorn7]

So first the good news: The AMD64 build still works. It fails only with the 2nd architecture it tries to build, which is armv7. The first failed build is from Dec 18, and that's after https://quay.io/repository/prometheus/busybox-linux-armv7 switched to SHA256 b5eff00f4a71 (see history).

It looks like the new busybox build does not support mkdir anymore.

I don't understand how this image is built, because the Quay repo doesn't show any builds.

I'm also very confused that the AMD64 build still seems to have no problems with mkdir.

@SuperQ you seem to have done some things in December with our busybox stuff. Maybe any of this makes sense to you?

[SuperQ]

Yes, we finally fixed the busybox builds that had been failing for many months. I'm not sure how this would affect mkdir directly. But it's possible that the architecture change broke something.

[beorn7]

@SuperQ applied a tricky work-around here.

As far as I can see, the build succeeded and we can close this.

[jeesmon]

Awesome, thanks @SuperQ and @beorn7

[jeesmon]

@SuperQ I see this in the build log for linux-armv7. linux-amd64 worked fine.

[beorn7]

But why does the CI consider the build as successful?

Anyway, the busybox on armv7 seems to be pretty crippled… 🤔

[beorn7]

I poked around a bit, but couldn't find another repo where we still use CircleCI and also use chown or mkdir.

For Alertmanager, we use chown and mkdir but the build process runs git GH actions. It seems to work fine for armv7 (I looked at https://github.com/prometheus/alertmanager/actions/runs/21710431110/job/62612851583 ).

I'm not familiar enough with Docker builds to spot the differences here. The busybox image pulled is the same for Alertmanager and Pushgateway ( quay.io/prometheus/busybox-linux-armv7:latest@sha256:debca5671... ).

[beorn7]

I asked Gemini to tell me why it's failing on CircleCI and succeeding on GH actions.
Their answer:

The issue you are encountering is a classic "system call mismatch" typically seen when running containerized builds for foreign architectures (like armv7) on a host with an older kernel or an outdated version of QEMU.

The Root Cause: System Call Incompatibility
The error chown: Function not implemented and the exit code 127 indicate that the system call required by the chown binary in the new Ubuntu 24.04-based busybox image is not being correctly translated by the emulation layer.

Even though you are using the "same" image as the alertmanager build, the environments differ significantly:

Log 1 (GitHub Actions): Running on a very modern 6.11.0-1018-azure kernel with a fresh installation of tonistiigi/binfmt (QEMU v10.0.4).

Log 2 (CircleCI): Likely running on an older host kernel or using the default CircleCI QEMU setup, which may not support the newer statx or chown system calls required by binaries compiled against newer GLIBC versions.

Why it Fails on armv7 specifically
Modern Linux distributions (and the binaries inside them) have transitioned to using the statx system call and 64-bit time types. Older versions of QEMU do not know how to map these calls from the armv7 (32-bit) container to the x86_64 (64-bit) host.

Recommended Solutions

1. Update the Emulation Layer (Preferred)
   If your CircleCI configuration allows it, ensure you are pulling the latest binfmt/QEMU binaries before starting the build. In your CircleCI config, add or update the step that sets up QEMU:

YAML

• run:
  name: Update QEMU
  command: docker run --privileged --rm tonistiigi/binfmt --install all

This matches the successful setup seen in your GitHub Actions log.

This sounds plausible (but AI is good in giving plausible looking answers…). The new busybox build running against a potentially very old Glibc.

I have looked at our CircleCI setup, and it feels we should probably update stuff over in https://github.com/prometheus/circleci . @simonpasquier you are listed as a maintainer over there. Could you perhaps double-check if the answer above makes sense and if there is anything we can easily do in https://github.com/prometheus/circleci ?

Of course, this might also be used as a reason to consolidate everything on GH actions, so we have to do this kind of housekeeping only once. (Who did the migration back then? @roidelapluie ?)

[SuperQ]

Ok, publish now works on GitHub Actions.

https://github.com/prometheus/pushgateway/actions/runs/21922534576

[beorn7]

Can we completely remove CircleCI now? How to do that cleanly?