Skip to content

zola, tamarin-prover: fix reproducibility (parallel codegen order; compile-time clock) - #525

Draft
bryan-minimal wants to merge 2 commits into
mainfrom
bryan/zola-tamarin-repro
Draft

zola, tamarin-prover: fix reproducibility (parallel codegen order; compile-time clock)#525
bryan-minimal wants to merge 2 commits into
mainfrom
bryan/zola-tamarin-repro

Conversation

@bryan-minimal

Copy link
Copy Markdown
Member

Both packages were flagged unknown by a build-twice audit of the 87 packages added since the last fleet run (83/86 reproducible, 96.5%). They looked identical in the verdict table and turned out to be completely different problems.

zola — the reproducibility guide was applied halfway

The recipe already cites minimal-repro's guide, and applies the path half:

# Reproducibility (per minimal-repro's guide): strip absolute build
# paths (source dir + cargo registry) and disable incremental builds.
export RUSTFLAGS="-C linker=gcc --remap-path-prefix=... --remap-path-prefix=..."
export CARGO_INCREMENTAL=0

…but never pinned codegen. rustc's default release build shards codegen across parallel units, and those units finish in whatever order the thread pool produces — so functions are emitted in a different order every build.

The forensics say exactly that, independently of reading the recipe:

  • 16.89% of bytes differ, while the total size is identical
  • 69% of sampled differing windows have a byte-exact twin elsewhere in the other build

A size-preserving permutation is ordering non-determinism, not codegen variance. Fix:

  • -C codegen-units=1 — one unit, one deterministic emission order
  • -C symbol-mangling-version=v0 — the legacy mangling scheme embeds a per-compilation-session hash in symbol names

Same fix that landed for nushell (#292) and difftastic.

tamarin-prover — not GHC, and not the toolchain

Worth stating plainly, since the GHC determinism work is recent: that work is holding. tamarin's Haskell codegen is bit-identical across builds. What differs is 26 bytes out of a 135 MB binary:

Git revision: UNKNOWN, branch: UNKNOWN
Compiled at: 2026-07-25 03:47:54.541898723 UTC     <- build A
Compiled at: 2026-07-25 03:53:20.855637052 UTC     <- build B

tamarin's version banner embeds the wall-clock compile time through a TemplateHaskell splice that calls getCurrentTime while compiling. It asks the clock directly, so the sandbox's SOURCE_DATE_EPOCH never reaches it. (The git fields are already deterministic — there's no repo in the build tree, so both builds say UNKNOWN.)

This is the same family as perl's cf_time (#291): an application baking a timestamp, not a compiler being nondeterministic.

The patch rewrites the splice to a fixed instant derived from SOURCE_DATE_EPOCH, and locates the source file by content (grep -rl 'Compiled at') rather than by path, so an upstream file move fails loudly here instead of silently reverting to a wall clock. It follows the verify-grep style the recipe already uses for its other source patches.

Verification status — please read

  • tamarin-prover: the mechanism is confirmed from the built artifacts (the two differing timestamp strings above, with everything else byte-identical). The patch itself has not yet been through a build-twice — tamarin is a multi-hour Haskell build.
  • zola: a double-build was started via the new min build path and the client detached mid-build, so I do not have a byte-identical result to show yet.

Marking this a draft for that reason. This repo's rule has been "never claim fixed without a real build-twice", and I'd rather show the analysis than assert a result I haven't seen. I'll post both double-build results here before marking it ready.

Reviewers: the reasoning is independently checkable from the measurements above even before the builds land.

Part of the reproducibility epic: gominimal/inbox#253

🤖 Generated with Claude Code

Both were flagged `unknown` by a build-twice audit of the 87 packages added
since the last fleet run. They turned out to be completely different problems.

zola — the reproducibility guide was applied halfway. The recipe already cites
minimal-repro's guide and strips build paths, but never pinned codegen. rustc's
default release build shards codegen across parallel units that finish in
thread-completion order, so functions are EMITTED in a different order each
build. Measured on 0.22.1: 16.89% of bytes differed while the total size stayed
identical, and 69% of differing windows had a byte-exact twin elsewhere in the
other build — a size-preserving permutation, i.e. ordering, not codegen
variance. Adds `-C codegen-units=1` (ordering) and
`-C symbol-mangling-version=v0` (legacy mangling embeds a per-session hash).
Same fix as nushell and difftastic.

tamarin-prover — not GHC, and not the toolchain. Its version banner embeds the
wall-clock compile time via a TemplateHaskell splice that calls getCurrentTime
while compiling; it asks the clock directly, so SOURCE_DATE_EPOCH never reaches
it. Exactly 26 bytes of a 135 MB binary differ, and the Haskell codegen is
otherwise bit-identical — the GHC determinism work is holding fine. Rewrites the
splice to a fixed instant derived from SOURCE_DATE_EPOCH, locating the source
file by content so an upstream file move fails loudly instead of silently
reverting to a wall clock.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 265f7123-6d11-4244-bcab-2c4a6f4ec135

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch bryan/zola-tamarin-repro

Comment @coderabbitai help to get the list of available commands.

The reproducibility patch only matched a BARE call:

    s|runIO Data\.Time\.getCurrentTime|...|
    s|runIO getCurrentTime|...|

but upstream composes the action rather than naming it:

    $(stringE =<< runIO (show `fmap` Data.Time.getCurrentTime))

`runIO` is followed by `(show ` fmap ` ...`, so neither pattern matched,
getCurrentTime survived, and the guard correctly refused the build on both
arches rather than shipping a wall-clock binary:

    ERROR: tamarin compile-time-clock patch did not apply in src/Main/Console.hs

Add a pattern for the whole parenthesised argument; keep the two bare forms in
case upstream simplifies back to them. Verified by running the patched block
verbatim against the real line:

    compileTime = "Compiled at: " ++ $(stringE =<< pure ("1970-01-01 00:00:00 UTC"))

which type-checks (`pure :: String -> Q String` feeding `stringE`) and leaves no
getCurrentTime for the guard to catch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants