Distroless ci-utils (feature/distroless)

[AndrewGearhart]

Overview

Create a Distroless variant of the ci-utils image (branch feature/distroless) that will package only the runtime CLI tools needed in CI while reducing image size and preserving CircleCI compatibility. Investigate splitting Trivy out of the main image so vulnerability scanning will be provided by the official Trivy container (aquasec/trivy) instead of embedding the large binary.

---

Branch / files

• Branch: feature/distroless
• Key files:
  • Dockerfile — original Ubuntu-based build; will be updated to use robust curl flags and strip downloaded binaries.
  • Dockerfile.distroless — multi-stage builder that will pack selected binaries and their shared libraries into a gcr.io/distroless/cc final image and will ensure /bin/sh is available (symlinked to bash).
  • bin/ — repository helper scripts copied into the images at build time.

---

Current branch contents (what exists on feature/distroless)

• Robust download logic for external CLI artifacts (curl retries, timeouts, validation).
• Binary stripping (strip --strip-all) applied where safe to reduce size.
• A packing step that collects binaries and their ldd dependencies into /pack and copies them into the Distroless final image.
• Verified builds for --platform=linux/amd64; smoke-tested presence of git, Docker CLI, buildx, yq, trivy, and a POSIX shell.

---

Motivation

• Distroless final images reduce attack surface and image size. However, dynamically linked CLI tools bring many shared libraries.
• The largest single artifact observed in the packed filesystem is the embedded trivy binary (~160MB), which dominates final image size.
• Running Trivy via the official container image (aquasec/trivy) will avoid embedding that large binary and will allow ci-utils to remain smaller and more focused.

---

Proposed work (tasks to complete on feature/distroless or follow-up branches)

1. Convert feature/distroless into the canonical Distroless build for ci-utils: finalize the packed list of runtime tools and include only what CI jobs require in the final image (for example: git, Docker CLI, buildx, yq, bash, curl).
2. Remove the embedded trivy binary from both Dockerfile and Dockerfile.distroless. Update documentation and CI workflows to invoke Trivy via the official container image:

   docker run --rm aquasec/trivy:latest image --severity HIGH,CRITICAL alpine:3.18

3. Add an opt-in vendor path for offline use: support a build-time flag (e.g., VENDOR_TRIVY=1) which will COPY a vendored Trivy tarball into the image during offline/enterprise builds.
4. Add automated verification: a CI job matrix that:
   • Builds ci-utils:distroless.
   • Runs smoke tests (verify git, docker, buildx, yq, /bin/sh).
   • Runs Trivy via aquasec/trivy against a known test image to confirm parity.

---

Acceptance criteria

• ci-utils:distroless builds reproducibly and contains required runtime tools with no missing shared libraries and a working /bin/sh compatible with CircleCI executors.
• ci-utils images will no longer contain the embedded trivy binary by default (unless VENDOR_TRIVY=1 is explicitly used) and final image sizes will be reduced measurably.
• CI documentation and pipelines will call Trivy via aquasec/trivy (or use the vendored flag) with equivalent results to the previous embedded workflow.

---

Validation steps (commands / checks to include in the ticket)

• Build Distroless image:
  docker build --platform=linux/amd64 -f Dockerfile.distroless -t ci-utils:distroless .

  • Smoke test inside image:
    docker run --rm --platform=linux/amd64 ci-utils:distroless git --version
    docker run --rm --platform=linux/amd64 ci-utils:distroless /usr/local/bin/yq --version
    docker run --rm --platform=linux/amd64 ci-utils:distroless /bin/bash -c 'echo shell-ok'

  • Demonstrate Trivy usage via official image (replacement for embedded binary):
    docker run --rm aquasec/trivy:latest image --severity HIGH,CRITICAL alpine:3.18

  • Produce a /pack inventory and sizes to show effect of removing Trivy:

          # inside the builder container or from a packed directory
          du -sh /pack
          find /pack -type f -printf '%s %p\n' | sort -nr | head -n 30

---

Follow-ups / optional optimizations

• Further strip binaries and consider static builds for tools where available to avoid pulling large libc/glibc footprints.
• Optionally split ci-utils into smaller, single-purpose images (e.g., ci-utils-core, ci-utils-security) if multi-image job orchestration is acceptable in CI to reduce per-job pull sizes.

---

Branch: feature/distroless — track finishing the Distroless image work, removing/externally-providing Trivy, adding the vendoring option, and adding CI verification.