How isolated are builds from each other?

[natevw]

The docs for container build (src) say that:

The build runs in isolation using BuildKit, […]

But if I set up two Dockerfiles with different steps (e.g. RUN sleep 42 vs. RUN sleep 45…) and build them in parallel, then when I do container ls, I still only see one "instance" of the buildkit(ghcr.io/apple/container-builder-shim/builder) container running. And furthermore when the build(s) complete this instance tends to stay running.

So with regards to building untrusted and/or outright malicious Dockerfiles is seems safe to say my macOS host itself is fairly isolated from any shenanigans any and all builds might try. The build happens within a VM, so barring any hypervisor/hardware-level escapes the build can only access the host via the shim itself. Thus any risks to the host would typically be limited to e.g. exfiltration via intentional COPY-type features, though possibly could exploit any unintentional vulnerabilities present in the shim and/or its client I suppose.

But on the flip side, is it not safe to assume that builds are isolated from each other? Just given the way the buildkit container appears to be handled, a malicious build would not have to escape any VM [or even container?] boundary to interfere with a simultaneous build or persist states of mayhem against future builds happening within that long-lived container, is that correct?

It seems that to isolate builds from each other and not just the host, I should:

1. try to avoid building more than one image at a time, and then
2. reset using container rm -f buildkitcontainer builder delete -f between each build

Would those precautions indeed be necessary to isolate untrusted builds from each other? (And if so, should they also be sufficient?)

[jglogan]

I’m not a buildkit expert so someone can correct me if this isn’t right, but buildkit uses the same Linux isolation techniques that regular Docker containers do - namespaces, cgroups, root pivoting and so on.

Technically we could run builds in individual builder VMs and that would provide stronger isolation. Average performance would be lower and disk capacity utilization would be higher as each builder would need to warm and retain its own layer content cache.

[natevw]

I’m not a buildkit expert so someone can correct me if this isn’t right, but buildkit uses the same Linux isolation techniques that regular Docker containers do - namespaces, cgroups, root pivoting and so on.

Me neither, but thanks for confirming there is at least some manner of container boundary still involved. What that means is still somewhat nebulous to me as there are a lot of variables and factors (and default capabilities) involved with containers in the first place. Beyond just the perennial question of whether containers "are secure" or only iff you hold them right, for the build case the in-band RUN --security=insecure … Dockerfile syntax gives me particular pause.

Technically we could run builds in individual builder VMs and that would provide stronger isolation. Average performance would be lower and disk capacity utilization would be higher as each builder would need to warm and retain its own layer content cache.

Yeah, the additional VM layer makes isolation things a lot easier to reason about. In the build case I've always been even more paranoid just given the sort of open-ended nature of Dockerfiles (e.g. I knew about COPY/ADD of course but didn't learn about the various RUN security/device/mount/etc. options until more recently, and while those are the only that stand out as I re-skim the table of contents I really don't know what else the build [or its parent…] is able to opt itself into…)

Now granted I'll be the first to admit that my paranoia might be irrational (or more specifically: irrelevant). Why go to all this trouble to "secure" the (much more easily auditable) build process. And then remote in via how much VS Code yolo'ing who knows how many hundreds of megabytes of node.js libraries between the guest and host "boundaries"………?

All this to say, I certainly don't have strong opinion and am not currently knowledgeable/qualified enough to weigh in. The reuse of the buildkit instance seems reasonable. As a default I don't think it's any worse than how other container platforms handle their builds, presumedly.

I do wonder perhaps if a container build --singular option that spins up a new and ephemeral buildkit-${randid} instance dedicated to just that one build would be a good feature? I like the idea, though it leaves me with a nagging doubt when to use it:

• for builds one doesn't trust? (i.e. so as not to be able to contaminate future builds)
• or for builds one needs to be trustworthy? (i.e. so as not to be contaminated by previous builds)
• all the time? since who knows what is/isn't and does/doesn't need to be trusted and less decision fatigue to just always? 🤷