fix(libvirt): make boot artifacts deterministic, portable, and cache-safe

[doublewhy]

Gap Claim

A local reliability audit of the dev reference implementation found three coupled boot-artifact failures:

• BusyboxInitramfsBuilder and GuestObservingInitramfsBuilder default to /usr/bin/busybox, while _cpio_newc delegates archive construction to an ambient host cpio. The default suite therefore fails on supported non-Ubuntu development hosts before the intended assertions run.
• identical appliance inputs do not produce identical initramfs bytes because host cpio retains ambient metadata and gzip.compress records wall-clock time;
• copy_kernel_for_libvirt treats equal st_mtime_ns values as content identity, so a changed kernel with a preserved timestamp returns stale cached bytes.

Evidence: implementations/python/packages/raes_backend_libvirt/techvault_appliance.py, guest_appliance.py, and techvault_matrix.py; current dev default tests fail in the native/guest-certified modules on macOS. The blocked assurance claim is that the configuration-bound libvirt backend can reproduce, identify, and safely reuse the exact boot artifacts it claims.

Existing Surface Audit

Checked:

• libvirt driver/interpreter, artifact binding, TechVault lifecycle, guest-certified observation, envelope/configuration artifacts, CLI, real-daemon harness, and ASR-519 conformance tests;
• SDL nodes/resources/content/source semantics, plan carriers, runtime snapshot/evidence carriers, published schemas, module aliases/imports, examples, docs, ADRs, and issues 01: Publish configuration-specific libvirt realization envelopes (ASR-519) #100, Scaffold a libvirt provisioning backend from the in-memory reference #601-606, 02: Enforce honest TechVault appliance realization disclosure (ASR-519) #714-717;
• recent dev work for node architecture and resolved-realization authority, plus PR fix(libvirt): reject shell injection via guest init-script interface fields #1085 for adjacent init-script safety.

This is not a new SDL or schema concept. The owning boundary is the non-normative libvirt adapter's boot-artifact builder and cache. Scenario fields, generic metadata, and downstream evidence-only patches cannot make nondeterministic or stale bytes correct.

Lineage and Precedent

• ASR-519 requires configuration-bound, reproducible realization evidence and no silent substitution.
• Issue 02: Enforce honest TechVault appliance realization disclosure (ASR-519) #714 requires requested values/artifacts to be realized, rejected, or explicitly transformed; issue 05: Certify the libvirt reference scenario with honest evidence (ASR-519) #717 requires reproducible clean runs and bound boot-artifact digests.
• RUN-314 and ADR-063 establish repository-owned reference implementation behavior rather than ambient-host interpretation.
• PR fix(libvirt): reject shell injection via guest init-script interface fields #1085 hardens the same init-script boundary and will be integrated without its unrelated .DS_Store artifact.

Literature and Practice

• Linux documents the initramfs newc wire format as a stable kernel interface: https://www.kernel.org/doc/html/latest/driver-api/early-userspace/buffer-format.html
• Python exposes an explicit gzip mtime control for reproducible output: https://docs.python.org/3/library/gzip.html#gzip.compress
• OCI descriptors define content identity by digest, not filesystem timestamps: https://github.com/opencontainers/image-spec/blob/main/descriptor.md
• Reproducible Builds documents normalized archive metadata and timestamps as necessary inputs: https://reproducible-builds.org/docs/archives/

Alternatives

1. Do nothing / document Ubuntu-only prerequisites. Rejected: the package declares Python portability, unit tests should be hermetic, and nondeterministic digests remain incorrect even on Ubuntu.
2. Keep host cpio and add flags/environment normalization. Rejected: supported cpio implementations differ, the current newline protocol is fragile, inode metadata remains ambient, and tool discovery still changes behavior.
3. Use mtime plus size for the kernel cache. Rejected: both are mutable metadata and can collide for different content.
4. Chosen: a small repository-owned newc encoder with fixed metadata/order and gzip mtime=0; explicit/discovered executable configuration with typed preflight; content-digest-validated, atomic kernel cache writes.

Chosen Architecture

• Encode the bounded appliance tree directly in canonical newc: sorted POSIX paths; fixed inode sequence, uid/gid/mtime/device fields; explicit regular/directory/symlink modes; four-byte alignment; trailer.
• Separate executable discovery/preflight from archive bytes. Callers may inject a static BusyBox path; defaults discover supported candidates and return a stable libvirt diagnostic before native mutation if missing or unsuitable.
• Cache copied kernels by actual content digest. Write a sibling temporary file, set the intended readable mode, and atomically replace the cache target only after the complete copy is durable enough for the reference adapter's stated contract.
• Preserve the existing public plan/schema surface. No new normative authority is introduced.

Documentation Defense

Add a remediation/preflight decision record explaining the owning boundary, deterministic format, rejected host-tool and metadata-cache alternatives, supported toolchain assumptions, and the limits of reproducibility (kernel bytes and appliance inputs remain external declared inputs). Update operator/reference docs where prerequisites or evidence interpretation change.

Verification Plan

• byte-identical repeated builds, including separate roots and changed wall-clock time;
• a parser-level assertion over newc entries, modes, contents, links, alignment, and trailer;
• missing/non-static/injected BusyBox preflight tests on all hosts;
• same-mtime/different-content kernel regression, no-op same-content reuse, atomic replacement/failure tests;
• existing TechVault/guest-certified focused tests and PR fix(libvirt): reject shell injection via guest init-script interface fields #1085 injection regressions;
• lint, repo policy, requirement governance, tools/verify_all.py, and the canonical verification graph;
• trace implementation/tests/docs to RUN-314 and ASR-519.

Implementation branch: RUN-314-libvirt-reliability.

[doublewhy]

Implemented and verified in merged PR #1128. Closing manually because the PR merged into the dev branch, while GitHub auto-closes linked issues only when changes reach the default branch.