[API Proposal]: Add reproducible/deterministic mode to TarWriter for normalized metadata

[jetersen]

Background and motivation

Work on dotnet/sdk#55689 exposed two separate sources of non-reproducible data when creating PAX archives with TarWriter:

1. Process-dependent PAX header names. TarHeader.GenerateExtendedAttributeName() writes names such as <directory>/PaxHeaders.<process-id>/<file>, and global extended headers use <tmp>/GlobalHead.<process-id>.<sequence>. These names are serialized into the tar stream, so writing the same PaxTarEntry in two different processes produces different archive bytes.
2. Host filesystem metadata captured by the path overload. On Unix, TarWriter.WriteEntry(string sourcePath, ...) reads mtime, mode, uid, gid, uname, and gname from the source filesystem. The ownership fields vary between machines and builder accounts even when the file contents are identical.

These are related but distinct behaviors.

The SDK PR does not use WriteEntry(string sourcePath, ...) for container layers. Layer.FromDirectory manually constructs PaxTarEntry instances, sets their timestamps and modes, opens files only for their content, and calls WriteEntry(TarEntry). Manually constructed POSIX entries already default to uid=0, gid=0, uname="", and gname="", so the builder account is not captured by that code path. The write-through stream in dotnet/sdk#55689 specifically normalizes the process-dependent PaxHeaders.<pid> name.

The runtime API should therefore cover both TarWriter-owned sources of nondeterminism:

• use process-independent names for PAX extended and global extended headers;
• provide deterministic defaults for metadata captured by WriteEntry(string sourcePath, ...);
• allow callers to provide a stable modification timestamp, commonly derived from SOURCE_DATE_EPOCH;
• retain explicit caller-provided metadata when writing a manually constructed TarEntry.

Entry order, file content, links, and file modes remain intentional inputs controlled by the caller. TarWriterOptions.HardLinkMode already controls whether filesystem hard-link identity affects the archive.

In OCI image scenarios, each layer tar archive is hashed. Any process ID, host ownership value, or filesystem timestamp serialized into the archive changes the layer digest and prevents registries from deduplicating otherwise identical layers.

API Proposal

namespace System.Formats.Tar;

public sealed class TarWriterOptions
{
    // Existing properties
    public TarEntryFormat Format { get; set; }
    public TarHardLinkMode HardLinkMode { get; set; }

    /// <summary>
    /// Gets or sets whether TarWriter should avoid process- and host-dependent metadata.
    ///
    /// When enabled:
    /// - PAX extended and global extended header names do not contain the process ID or TMPDIR.
    /// - Entries created by WriteEntry(string sourcePath, ...) use uid=0, gid=0,
    ///   uname="", and gname="", unless overridden below.
    /// - Entries created by WriteEntry(string sourcePath, ...) use
    ///   OverrideModificationTime, or UnixEpoch when no override is provided.
    ///
    /// Explicit metadata on entries passed to WriteEntry(TarEntry) is preserved. The
    /// process-independent PAX header naming still applies to those entries.
    /// </summary>
    public bool Deterministic { get; set; }

    /// <summary>
    /// Gets or sets the modification timestamp used for entries created from filesystem paths.
    /// In deterministic mode, null means DateTimeOffset.UnixEpoch.
    /// Outside deterministic mode, null preserves the source filesystem timestamp.
    /// </summary>
    public DateTimeOffset? OverrideModificationTime { get; set; }

    /// <summary>
    /// Gets or sets the user ID used for entries created from filesystem paths.
    /// In deterministic mode, null means 0.
    /// Outside deterministic mode, null preserves the source filesystem value.
    /// </summary>
    public int? OverrideUid { get; set; }

    /// <summary>
    /// Gets or sets the group ID used for entries created from filesystem paths.
    /// In deterministic mode, null means 0.
    /// Outside deterministic mode, null preserves the source filesystem value.
    /// </summary>
    public int? OverrideGid { get; set; }

    /// <summary>
    /// Gets or sets the user name used for entries created from filesystem paths.
    /// In deterministic mode, null means an empty string.
    /// Outside deterministic mode, null preserves the source filesystem value.
    /// </summary>
    public string? OverrideUName { get; set; }

    /// <summary>
    /// Gets or sets the group name used for entries created from filesystem paths.
    /// In deterministic mode, null means an empty string.
    /// Outside deterministic mode, null preserves the source filesystem value.
    /// </summary>
    public string? OverrideGName { get; set; }
}

The exact property names are open for discussion. The important part of the proposal is that deterministic mode covers the PAX header name generated internally by TarWriter, not only ownership metadata captured by the filesystem-path overload.

API Usage

// Example 1: Reproducible archive directly from filesystem paths
using var layerStream = new MemoryStream();
var options = new TarWriterOptions
{
    Format = TarEntryFormat.Pax,
    Deterministic = true,
    OverrideModificationTime = DateTimeOffset.FromUnixTimeSeconds(sourceDateEpoch)
};

using (var writer = new TarWriter(layerStream, options, leaveOpen: true))
{
    writer.WriteEntry("/app/config.json", "config.json");
    writer.WriteEntry("/app/app.exe", "app.exe");
}

// The source file modes are preserved, while process ID, builder ownership and
// filesystem mtime no longer affect the archive bytes.

// Example 2: SDK-style manually constructed entries
using var layerStream = new MemoryStream();
var options = new TarWriterOptions
{
    Format = TarEntryFormat.Pax,
    Deterministic = true
};

using (var writer = new TarWriter(layerStream, options, leaveOpen: true))
{
    var entry = new PaxTarEntry(TarEntryType.RegularFile, "app.exe")
    {
        DataStream = File.OpenRead("/app/app.exe"),
        ModificationTime = DateTimeOffset.FromUnixTimeSeconds(sourceDateEpoch),
        Uid = configuredContainerUid
    };

    writer.WriteEntry(entry);
}

// Explicit entry metadata is retained, but the generated PAX extended-header name
// is stable and no post-processing stream is required.

// Example 3: Custom ownership for path-based writes
var options = new TarWriterOptions
{
    Format = TarEntryFormat.Pax,
    Deterministic = true,
    OverrideModificationTime = DateTimeOffset.FromUnixTimeSeconds(sourceDateEpoch),
    OverrideUid = 0,
    OverrideGid = 0,
    OverrideUName = "root",
    OverrideGName = "root"
};

Alternative Designs

Design 1: Boolean deterministic mode plus override properties (proposed)

• One switch covers TarWriter-generated process-dependent names and deterministic defaults for path-based writes.
• Override properties support SOURCE_DATE_EPOCH, root ownership, and other reproducible policies.
• Explicitly constructed TarEntry metadata remains under caller control.

Design 2: Enum-based metadata mode

public enum TarMetadataMode
{
    Preserve,
    Deterministic,
    NormalizeToRoot,
    Custom
}

A separate option would still be needed for the modification timestamp, and the deterministic mode would still need to define process-independent PAX header naming.

Design 3: Separate options for each behavior

public bool UseDeterministicPaxHeaderNames { get; set; }
public bool NormalizeFileSystemOwnership { get; set; }
public DateTimeOffset? OverrideModificationTime { get; set; }

This is more explicit, but makes the common reproducible-archive scenario easier to configure incompletely.

Design 4: SOURCE_DATE_EPOCH environment-variable support

TarWriter could read SOURCE_DATE_EPOCH directly and enable deterministic behavior automatically. This follows the reproducible-builds convention but introduces implicit environment-dependent library behavior. Passing the parsed timestamp through TarWriterOptions is more explicit.

Design 5: Continue constructing entries manually and post-process the stream

This is possible today and is what dotnet/sdk#55689 does. It works, but requires every caller to understand TarWriter's internal PAX naming and filesystem metadata behavior.

Risks

Breaking changes: None. Existing behavior remains the default. The proposed behavior is opt-in.

Performance: Deterministic path-based writes skip user and group database lookups. This should be neutral or faster.

Platform behavior:

• Windows path-based writes already default to uid=0, gid=0, empty uname/gname.
• Unix path-based writes change only when deterministic mode or an explicit override is selected.
• Process-independent PAX header naming applies consistently across platforms.

PAX-specific concerns:

• Both regular extended header names and global extended header names must be process-independent.
• Ownership values must be normalized in both standard header fields and PAX extended attributes when applicable.
• Explicit ownership on a manually constructed TarEntry must not be discarded.
• Tests should compare raw archive bytes across different process IDs and path-based source ownership values.

Timestamp concerns:

• WriteEntry(string sourcePath, ...) currently captures filesystem mtime, so ownership normalization alone cannot promise byte-identical output.
• A stable timestamp must be part of deterministic mode, either Unix epoch or an explicit value such as SOURCE_DATE_EPOCH.

Container registry implications:

• Enabling deterministic mode changes existing layer digests once.
• Subsequent builds from the same content and intentional metadata produce stable digests and can be deduplicated.

Compatibility with dotnet/sdk#55689:

• The SDK workaround was introduced for the process ID in generated PAX extended-header names.
• The SDK already manually constructs PaxTarEntry objects, so it does not capture the builder's uid, gid, uname, or gname through the path overload.
• Native process-independent PAX header naming would allow the SDK to remove PaxHeaderNameNormalizingStream.
• The path-based ownership and timestamp options remain valuable for TarFile.CreateFromDirectory and other callers that use filesystem-path APIs.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.