containerd daemon: optional publish-then-pull workaround to avoid dig…

[casibbald]

Summary

When the Docker daemon uses containerd storage (default on Docker Desktop and Colima), exporting the built app image directly to the daemon can fail with digest errors or be severely degraded:

• Performance: Export to the daemon can take ~9 seconds instead of milliseconds, and there can be a ~20s delay before the "Analyzing" phase (#2272).
• Failures: Saving the image to the daemon can fail with unable to create manifests file: NotFound: content digest sha256:...: not found (#2270, closed as known limitation).

This PR adds an optional workaround when building without --publish:

1. Detect containerd-backed daemons via docker info (driver-type = io.containerd.snapshotter.v1).
2. When not publishing, not using layout, and no previous image: publish the app image to a local registry (default localhost:5001), then pull it into the daemon and tag it as the user-requested image. The lifecycle thus pushes to the registry instead of writing to the daemon, avoiding the digest path.
3. Registry is configurable via PACK_CONTAINERD_WORKAROUND_REGISTRY (e.g. host.docker.internal:5001 if the registry runs on the host). The workaround registry is automatically added to insecure registries for the lifecycle when needed.
4. When the workaround does not apply (e.g. layout build, or previous image set) but the daemon still uses containerd, we warn that export may be slow or fail and link to pack build takes 27x longer when using containerd image store #2272.

Scope: Only applies when opts.Publish is false. When the user passes --publish, pack already pushes to the registry and no workaround is used.

Output

Before

With containerd storage and pack build myimage (no --publish):

• Build could fail with digest error, e.g. unable to create manifests file: NotFound: content digest sha256:...: not found (see Build fails if containerd is used with an untrusted builder #2270).
• Or export could take ~9 seconds and there could be a ~20s delay before Analyzing (see pack build takes 27x longer when using containerd image store #2272).

No automatic workaround; user had to either disable containerd, use pack build --publish to a registry, or run a local registry and use --publish to it.

After

With containerd storage and pack build myimage (no --publish), and no layout/previous image:

• Pack detects containerd and enables the workaround.
• Log message: Daemon uses containerd storage; using publish-then-pull workaround (registry: localhost:5001). See https://github.com/buildpacks/pack/issues/2272.
• Lifecycle publishes the app image to the configured registry (default localhost:5001); pack then pulls it into the daemon and tags as myimage. Build completes without writing the app image directly to the daemon.
• User must have a registry reachable at the workaround address (e.g. docker run -p 5001:5000 registry:2 for default). Override with PACK_CONTAINERD_WORKAROUND_REGISTRY=host.docker.internal:5001 if the registry runs on the host.

When workaround does not apply (e.g. layout or previous image) but daemon uses containerd:

• Warning: Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded. For more information, see https://github.com/buildpacks/pack/issues/2272.

Documentation

• Should this change be documented?
  • Yes, see #___
  • No (optional env var and behavior; can be documented in a follow-up if maintainers prefer)

Related

Resolves #2272

Related: #2270 (build failure with containerd + untrusted builder; same underlying daemon export / digest issue; closed with workaround guidance).

[jjbustamante]

Hi @casibbald

Thanks for this PR! I think the workaround of having a local registry and then pull it into the daemon should be documented and encourage people to use it, not sure about adding the hacking in the normal pack build flow. I mean, whoever wants to use the hack will need to explicitly have a local registry running and configure the env variable. Could we document those steps in our docs pack section?

1. Run a local registry
2. Run pack build using --publish flag pointing to that registry
3. Run docker pull to save it into the daemon

I know it is more comfortable to have pack doing that behind the scenes, but I would prefer to try to solve the original issue with containerd storage.