feat(image): implement optimized pure-Go native image rootfs export

[novahe]

This PR introduces a pure-Go, daemonless rootfs export mode (Native Export) that completely bypasses docker, skopeo, and umoci while maintaining industry-standard extraction performance.

This feature is opt-in and can be explicitly enabled by setting the CUBEMASTER_NATIVE_ROOTFS_EXPORT_ENABLED=true environment variable. By utilizing go-containerregistry and archive.Untar, the native pipeline fetches images and constructs the root filesystem natively in Go, providing a much lighter, faster, and more controllable alternative to the external CLI-based dockerless approach.

Key Features

1. Concurrent Prefetch Pipeline: Concurrently prefetches compressed layers to saturate network bandwidth, utilizing errgroup for immediate context cancellation and error propagation if any layer fails.
2. Phase 2 Loop-Mount Streaming: Unlocks direct stream extraction for Native mode. It bypasses intermediate host directories and extracts decompressed streams directly into the target ext4.img block device.
3. Aggressive Resource Cleanup: Implements a "decompress-and-delete" strategy that immediately removes compressed temporary files layer by layer, significantly minimizing peak disk space usage.

Comparison: Native vs. Dockerless

Core Advantage	Dockerless Mode (skopeo + umoci)	Native Mode (Pure-Go rootfs export)
I/O Pipeline	skopeo copy flushes the entire image to disk as an OCI layout first, followed by umoci unpack to extract the rootfs.	Concurrently prefetches compressed layers, decompresses layer by layer, and applies directly to the target directory. When the loop-mount fast path is enabled, it writes directly into the ext4 mount point.
External Dependencies	Rootfs export strongly depends on host skopeo and umoci subprocesses.	Rootfs export no longer depends on docker/skopeo/umoci subprocesses (Note: ext4 creation still depends on system tools).
Resource Control	External processes act as a black box. The temporary OCI layout + bundle/rootfs results in a significantly higher peak disk footprint.	Concurrency, cancellation, and layer-by-layer cleanup are controlled natively in Go. Peak disk usage comes strictly from the temporary compressed layer files and the target rootfs/ext4.

Test Results

Metric	Native	Dockerless	Advantage
Duration (min of 3 runs)	31.30s	37.26s	Faster by 5.96s (~16.0%)
Peak Memory (min of 3 runs)	22 MiB	36 MiB	Lower by 14 MiB (~38.9%)

Future Work

• Concurrency & Memory Control: Following this current work, we can now effectively control the concurrency of image layer fetching, which in turn enables us to throttle and manage the memory footprint of the process.
• Cross-Image Layer Caching: Explore supporting shared layer caches by dropping downloaded layer blobs into a local OCI-layout cache. This will enable layer sharing across different images, eliminating redundant blob downloads and significantly reducing network I/O.

Assisted-by: Codex:GPT-5

[cubesandboxbot]

Code Review: PR #558 -- Native Go rootfs export

Overall, this is a well-structured PR. The concurrent-download/sequential-extract design is correct for ordered OCI layers. Test coverage with testing/synctest is excellent.

Issue 1: PR description says opt-in but code is opt-OUT

File: native.go:38-42, PR description

The PR description states this feature is opt-in, but the function returns true when the env var is unset. Feature is enabled by default (opt-OUT). Only setting CUBEMASTER_NATIVE_ROOTFS_EXPORT_ENABLED=false disables it. Fix the description mismatch.

Issue 2: No progress feedback during native extraction

File: native.go:60-209

The dockerless path reports progress through OnPullProgress; the native path has no equivalent. Consider adding progress counters.

Issue 3: Silent env-var parse failures

File: native.go:41,226

Unparseable env values silently fall back to defaults. Log a warning when the env var is present but invalid.

Issue 4: convertV1Config cross-path compatibility undocumented

File: source.go:148-156

Document whether the DockerImageConfig output is compatible with the dockerless path.

Issue 5: Decompressor close errors silently discarded

File: native.go:195-196

Decompressor Close errors are swallowed. Log them.

Positive highlights

• Deterministic concurrency tests with testing/synctest

• Robust cancellation: context.AfterFunc + sync.Once (native.go:114-120)
• Same-filesystem temp dir (native.go:73)
• validateImageRef correctly bypassed for native mode (export.go:20-23)

[novahe]

PTAL @fslongjin

[kinwin-ustc]

@novahe,We are preparing for the release of the next version, and after it's sent out, we will take a look at this PR

[novahe]

@novahe,We are preparing for the release of the next version, and after it's sent out, we will take a look at this PR

OK, looking forward to your feedback as soon as possible.

[novahe]

Kindly ping @kinwin-ustc

[fslongjin]

Kindly ping @kinwin-ustc

I'll review this later today~

[fslongjin]

Review: Native rootfs export

Thanks for the PR. The overall direction matches the requirement: it adds an opt-in Native Export path controlled by CUBEMASTER_NATIVE_ROOTFS_EXPORT_ENABLED=true, resolves images with go-containerregistry, and applies OCI layers with containerd/archive without spawning docker/skopeo/umoci for rootfs export.

I do not think this is ready to merge yet. There are a few functional and correctness gaps that should be addressed first.

Blocking / high-risk issues

1. Native extraction does not currently have rootless semantics

   StreamRegistryToDir calls archive.Apply(ctx, destDir, decompressed) directly. containerd/archive.Apply preserves tar ownership by default, so in a non-root environment it tries to lchown files to UID/GID 0 and fails.

   I reproduced this locally:

   • go test ./pkg/templatecenter/image fails
   • Failing tests:
     • TestStreamRegistryWhiteoutResolution
     • TestStreamRegistryUsesPreparedNativeImage
   • Error: failed to Lchown ... operation not permitted

   This is both a test/CI issue and a production risk if Cubemaster runs without root or the required capabilities. The old dockerless path uses umoci unpack --rootless; the native path needs an equivalent decision. If rootless behavior is intended, please consider using archive.WithNoSameOwner() or another explicit rootless apply strategy. If root is required, the requirement should be documented and the tests should skip or split root-only coverage appropriately.

2. The advertised Phase 2 loop-mount streaming path is not active on the current main template-build flow

   BuildExt4 only uses loop-mount streaming when opts.PostRootfsExport == nil. However, buildRootfsArtifact always installs PostRootfsExport to bake the CubeEgress CA into the rootfs. That means the PR description's “Phase 2 Loop-Mount Streaming” behavior does not apply to the primary template build path today, even when there is no CA work to perform.

   If Phase 2 is part of this PR's core deliverable, the relationship between PostRootfsExport and streaming should be redesigned or explicitly narrowed. Otherwise, the PR description and tests should be updated so reviewers do not assume the primary path gets the streaming benefit.

3. The disk-space model is not quite “decompress-and-delete”

   The implementation first prefetches all compressed layers into prefetchDir, then sequentially decompresses/applies them and deletes each layer file after a successful apply. Peak disk usage therefore still includes all compressed layers plus the destination rootfs/ext4. For large images, this can still be significant.

   If minimizing peak disk usage is a goal, consider a bounded ordered pipeline that only keeps a limited window of downloaded layers and deletes them immediately after apply. If the current design is intentional, the PR description should describe the actual resource model more precisely.

Functional / architecture notes

• Replacing UseDockerless bool with ExportMode is a good direction because it keeps the prepare/export decision stable.
• PrepareLocalSource now goes to the remote registry when Native mode is enabled. This changes redo behavior from “local docker image must still exist” to “resolve/pull from registry again”, similar to dockerless. Please confirm redo always has the credentials needed for that path.
• Native currently hardcodes linux/$GOARCH via remote.WithPlatform(defaultPlatform()). That is fine for the immediate host-arch path, but it should be called out if cross-architecture template builds are expected later.
• PreparedSource now stores both nativeImage v1.Image and registry credentials. This is acceptable short term, but it makes the generic source object carry backend-specific execution state. If more backends or cache behavior are added, an exporter/backend abstraction may keep responsibilities cleaner.

Security notes

• The native path reduces CLI argument-injection exposure because image refs are no longer passed to docker/skopeo/umoci subprocesses. Keeping validateImageRef in prepare is still a reasonable defense-in-depth measure.
• Explicit registry credentials are passed through authn.FromConfig and are not written to disk, which is better than the dockerless authfile behavior.
• When no explicit credentials are provided, authn.DefaultKeychain may read the Cubemaster process user's Docker config. Please confirm this matches the intended multi-tenant boundary and document it if needed.

Test coverage

Good coverage added:

• Native enablement and concurrency env parsing
• Native prepare config/digest/size extraction
• PrepareSource / PrepareLocalSource native routing
• digest canonicalization
• whiteout / opaque-dir behavior
• reuse of the prepared native image
• context cancellation while scheduling downloads

Missing or currently weak coverage:

• Native extraction tests must pass in a normal non-root CI environment.
• Native + exportImageRootfs end-to-end path is not directly covered.
• Native + BuildExt4 primary Phase 1 path should be covered, including the PostRootfsExport case that currently disables streaming.
• Registry auth option behavior, default keychain behavior, private-registry failure paths.
• Multi-arch manifest/platform mismatch behavior.
• Cleanup behavior for prefetch temp files on download failure, apply failure, and context cancellation.
• Native export combined with CubeEgress CA baking.

Local verification

• go test ./pkg/templatecenter/image: failed due to the Native extraction lchown operation not permitted issue.
• go test ./pkg/templatecenter/...: failed for the same two Native extraction tests; pkg/templatecenter and pkg/templatecenter/cube_egress_ca passed.

Suggested merge checklist:

1. Fix or explicitly define rootless/owner-preservation behavior for archive.Apply, and make the new tests pass in normal CI.
2. Clarify or redesign the Phase 2 streaming interaction with PostRootfsExport.
3. Correct the prefetch/decompress/delete resource model or update the PR description.
4. Add Native + BuildExt4 coverage for both with and without PostRootfsExport.
5. Add registry auth, platform-selection, and failure-cleanup coverage.

[chenhengqi]

If we can do it in pure-Go, we can just retire all Docker/Dockerless code which requrires external CLIs.

[novahe]

If we can do it in pure-Go, we can just retire all Docker/Dockerless code which requrires external CLIs.

Yes, the goal is to replace them entirely. For now, we will enable native-export by default. Once it proves to be stable, we will delete the other code implementations.

If we encounter any issues, we can fall back to the previous behavior by setting CUBEMASTER_NATIVE_ROOTFS_EXPORT_ENABLED=false.

PTAL @fslongjin

[fslongjin]

This introduces a regression in the existing template image build progress reporting.

cubemastercli template watch and the TUI currently surface pull progress via pull_total_bytes, pull_downloaded_bytes, pull_speed_bps, and layer completion fields. Those fields are populated through the existing SourceSpec.OnPullProgress callback: the docker path reports through dockerPull, and the dockerless path reports through skopeoCopy.

In the new Native export path, however, progress reporting is not wired through. prepareNativeSource does not preserve spec.OnPullProgress on the returned PreparedSource, and StreamRegistryToDir downloads compressed layers via io.CopyBuffer without invoking the progress callback. As a result, when Native export is enabled, users can still see coarse job phase/progress updates, but the byte-level download progress, speed, and completed-layer details shown by cubemastercli regress or disappear.

Please keep the existing progress contract for the Native pipeline as well. A straightforward approach would be to use source.CompressedSizeBytes as TotalBytes, len(layers) as TotalLayers, accumulate downloaded bytes while each compressed layer is copied, update CompletedLayers when a layer finishes, and call source.OnPullProgress(image.PullProgress{...}). That should let the existing job progress sink and CLI rendering continue to work without CLI-side changes.

[novahe]

thank you for the reminder, I implemented it.

Please take a look again when you have time. @fslongjin