From 05b70224b70d4f9597ff17f622dcc02b219fc5f9 Mon Sep 17 00:00:00 2001 From: Reuben Brooks Date: Tue, 21 Jul 2026 16:10:08 -0500 Subject: [PATCH 1/2] test(divergence): add load-toplevel-echo cross-port case (pyrex41/shen-lua#40) Bifrost surfaced a real divergence found while running a Shen spec across shen-rust/shen-go/shen-lua: the canonical kernel load (klambda/load.kl shen.eval-and-print) prints each toplevel form's value before returning `loaded`. shen-rust and shen-go do; shen-lua does on a cold fasl cache / SHEN_FASL=off but DROPS the echo on a warm fasl-cache hit (documented degrade, boot.lua ~796). So (load file) stdout is cache-state dependent -- a cross-port divergence and an intra-port nondeterminism. - programs/load-echo-probe.shen: minimal probe ("PROBE" / (+ 40 2) / (define p)) - cases/divergences.json: load-toplevel-echo, mode eval, expect agreement + known_divergence shen-lua-fasl-load-echo (reports DIVERGE, not FAIL, until it converges -- then drop the marker for a hard agreement, as hush-file-write did) Verified: bifrost --only load-toplevel-echo across shen-rust,shen-go,shen-lua reports KNOWN DIVERGENCE: '"PROBE"\n42\n\nloaded' <- shen-go,shen-rust vs 'loaded' <- shen-lua. Filed pyrex41/shen-lua#40 with root cause + fix options. Co-Authored-By: Claude Opus 4.8 --- cases/divergences.json | 8 ++++++++ programs/load-echo-probe.shen | 17 +++++++++++++++++ 2 files changed, 25 insertions(+) create mode 100644 programs/load-echo-probe.shen diff --git a/cases/divergences.json b/cases/divergences.json index 8dce293..05ab447 100644 --- a/cases/divergences.json +++ b/cases/divergences.json @@ -6,6 +6,14 @@ "mode": "special-hush", "expect": "agreement", "doc": "RESOLVED (was a known divergence): under -q (*hush* = true), `pr` to a FILE stream now WRITES on every impl. *hush* only gates the standard output stream, never an explicit file stream. Previously shen-lua/shen-rust SILENCED the write (zero-byte file) under -q while shen-cl/shen-go/ShenScript wrote; the shen-lua (fix/hush-pr-file-22) and shen-rust (fix/hush-pr-file-2) fixes converged this. Now asserted as a HARD agreement: every available impl must land in the same bucket (all 'written'); any impl that silences the write is a real FAIL." + }, + { + "name": "load-toplevel-echo", + "mode": "eval", + "expr": "(load \"programs/load-echo-probe.shen\")", + "expect": "agreement", + "known_divergence": "shen-lua-fasl-load-echo", + "doc": "OPEN divergence. The canonical kernel load (klambda/load.kl -> shen.eval-and-print) PRINTS each toplevel form's evaluated value before returning `loaded`; this is the echo you see from `shen eval -l FILE` and `(load FILE)` at the REPL (NOT `script` mode, which is quiet on every port -- so use eval mode here). shen-rust and shen-go emit it; shen-lua emits it too on a COLD fasl cache or with SHEN_FASL=off, but on a WARM fasl-cache HIT it DROPS the per-form echo (boot.lua ~796; the documented 'a replayed load does not echo per-form values/types' degrade at boot.lua:476). Net effect: `(load file)` stdout is cache-state dependent -- a cross-port divergence (warm shen-lua vs everyone else) AND an intra-port nondeterminism (shen-lua cold != shen-lua warm). Scope: affects toplevel VALUE echo only; `(output ...)`/`(print ...)` side-effects are unaffected and agree everywhere. Minimal repro: eval `(load \"...load-echo-probe.shen\")` (file has \"PROBE\" / (+ 40 2) / (define p -> ok)); canonical prints PROBE, 42, (fn p) then loaded; warm shen-lua prints only loaded. Tracked: pyrex41/shen-lua#40 (fasl replay should re-emit the eval-and-print echo, or exclude the replay path from load's observable-output contract). Flip to a HARD agreement (drop known_divergence) once converged, as hush-file-write was." } ] } diff --git a/programs/load-echo-probe.shen b/programs/load-echo-probe.shen new file mode 100644 index 0000000..d3b3c76 --- /dev/null +++ b/programs/load-echo-probe.shen @@ -0,0 +1,17 @@ +\\ load-echo-probe.shen -- probe whether (load ...) echoes each toplevel form's +\\ evaluated value, as the canonical kernel load requires. +\\ +\\ klambda/load.kl: (defun shen.eval-and-print (Forms) +\\ (map (lambda F (pr (shen.app (eval-kl (shen.shen->kl F)) +\\ "\n" shen.s) (stoutput))) Forms)) +\\ i.e. load PRINTS each toplevel result before returning `loaded`. +\\ +\\ Canonical impls (shen-cl / shen-rust / shen-go, and shen-lua with a COLD fasl +\\ cache or SHEN_FASL=off) print: "PROBE" then 42 then (fn p). +\\ shen-lua on a WARM fasl-cache hit DROPS all three (boot.lua ~796; the +\\ documented "replayed load does not echo per-form values" degrade). That makes +\\ (load file) stdout depend on cache state -- a cross-port divergence and an +\\ intra-port nondeterminism. See cases/divergences.json load-toplevel-echo; pyrex41/shen-lua#40. +"PROBE" +(+ 40 2) +(define p -> ok) From d7f19f87c3592b28b96d416f659a3f41b5db1c8d Mon Sep 17 00:00:00 2001 From: Reuben Brooks Date: Tue, 21 Jul 2026 18:33:53 -0500 Subject: [PATCH 2/2] test(divergence): flip load-toplevel-echo to a hard agreement (#40) shen-lua's warm-fasl `(load)` echo divergence is fixed (pyrex41/shen-lua#40, PR #41): a warm cache hit now re-emits each top-level form's value, so `(load FILE)` stdout no longer depends on cache state and agrees with shen-rust/ shen-go cold and warm. Drop the `known_divergence` marker so the case is now a HARD agreement (a real FAIL if any port regresses), mirroring the lifecycle hush-file-write went through. Doc updated to RESOLVED with the PR ref. Also complete the run-time banner strip in normalize(): the kernel banner is `(cn "\nrun time: " ...)`, whose leading newline leaves a BLANK line ahead of the text line. We stripped only the text, so the residual blank diverged against a port that omits the whole banner (shen-lua on a warm hit re-emits the value echo but not the cosmetic banner). Drop that blank too, so the entire cosmetic banner is normalized away symmetrically across ports. README: refresh the divergence status (load-toplevel-echo now resolved; the list is all hard agreements) and a light intro trim. Verified: full corpus 31 pass / 0 diverge / 0 FAIL across shen-rust, shen-go, shen-lua, on both warm and cold shen-lua fasl caches. Co-Authored-By: Claude Opus 4.8 --- README.md | 41 +++++++++++++++-------------------------- bifrost.py | 12 ++++++++++-- cases/divergences.json | 3 +-- 3 files changed, 26 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index df49bdd..ed3c9e4 100644 --- a/README.md +++ b/README.md @@ -7,10 +7,8 @@ > behaviours across **all** the Shen implementations and asserts they produce > the same observable result (differential / conformance testing). -Bifrost sits alongside [Ratatoskr / Yggdrasil](../ratatoskr) in the same Norse -lineage — Ratatoskr is the squirrel that runs *up and down* the world-tree (the -two-stage shaker); Bifrost is the bridge that verifies the worlds at the ends of -the tree agree. +It sits alongside [Ratatoskr / Yggdrasil](../ratatoskr) — the two-stage shaker — +in the same Norse lineage, and drives it for the `--shake` deploy-path checks. Differential testing is its origin, but Bifrost is now three things: @@ -51,28 +49,19 @@ The corpus (`cases/*.json`, driven by `programs/*.shen`) includes: - **CLI parity** — `eval -e` prints the value; `(version)` / `--version` carry the kernel version **41.2**; **stdin-EOF causes a clean exit** (no hang) on every impl. -- **Known divergences** (documented, *not* hard failures): **none currently** — - all previously-tracked divergences have converged (see below). -- **Resolved divergences** (converged across all available impls; now asserted - as **hard agreements**, no longer tagged): - - ~~`float-formatting`~~ — **RESOLVED.** `(+ 0.1 0.2)` now prints the shortest - round-trippable form `0.30000000000000004` on *every* impl. shen-go - previously printed `0.300000` (`%f`, fixed in pyrex41/shen-go#11) and - shen-lua printed `0.3` (`%.14g` via `tostring`, fixed in - pyrex41/shen-lua#24). Exactly-representable floats such as `2.5` and `4.75` - also agree. Now asserted as a hard agreement — any impl that prints a - non-shortest-round-trip float is a real FAIL. - - ~~`int-div-zero`~~ — **RESOLVED.** `(/ 1 0)` now raises a *catchable* kernel - error on every impl, so `(trap-error (/ 1 0) …)` yields `divide-by-zero` - everywhere. shen-go previously returned `maxint` - (`9223372036854775807`); fixed on `fix-go-divzero-and-floatfmt`. Any impl - that fails to raise is now a real FAIL. - - ~~`hush-file-write`~~ — **RESOLVED.** Under `-q` (`*hush* = true`), `pr` to a - **file** stream now writes on *every* impl (`*hush*` gates only the standard - output stream). shen-lua (`fix/hush-pr-file-22`) and shen-rust - (`fix/hush-pr-file-2`) previously silenced the write (zero-byte file); both - fixed. Now asserted as a hard agreement — any impl that silences a - file-stream `pr` is a real FAIL. +- **Divergences** — **none open.** Every tracked cross-port difference has + converged and is now asserted as a **hard agreement** (a regression is a real + FAIL, no longer a documented difference): + - `float-formatting` — `(+ 0.1 0.2)` prints the shortest round-trippable + `0.30000000000000004` everywhere (was shen-go `0.300000` / shen-lua `0.3`; + pyrex41/shen-go#11, pyrex41/shen-lua#24). + - `int-div-zero` — `(/ 1 0)` raises a catchable `divide-by-zero` everywhere + (was shen-go `maxint`). + - `hush-file-write` — under `-q`, `pr` to a **file** stream writes everywhere + (`*hush*` gates only stdout; was a zero-byte file on shen-lua/shen-rust). + - `load-toplevel-echo` — `(load FILE)` echoes each top-level form's value + regardless of fasl-cache state (shen-lua dropped the echo on a warm cache + hit; pyrex41/shen-lua#40). - **Heavy** (`--heavy`) — **Ratatoskr stage-1 parity**: run `(ratatoskr.shake ["tests/fib.shen"] OUT)` on every host and assert the produced `kernel.kl` + `ratatoskr.manifest` are **byte-identical** across diff --git a/bifrost.py b/bifrost.py index 6c6781a..987ad99 100644 --- a/bifrost.py +++ b/bifrost.py @@ -252,8 +252,12 @@ def resolve(rel, default): def normalize(text, extra_prefixes=()): """Normalise an impl's stdout for comparison. - - strip a trailing run-time banner line ("run time: ... secs") emitted by - the kernel `load`/timer on several ports, + - strip the run-time banner ("run time: ... secs") emitted by the kernel + `load`/timer on several ports, INCLUDING the blank line the banner's + leading newline produces (`(cn "\nrun time: " ...)` -> a blank line then + the text line). Stripping only the text left the blank as residue, which + diverged against a port that omits the whole banner (e.g. shen-lua on a + warm fasl-cache hit re-emits the value echo but not the cosmetic banner), - strip `(fn NAME)` declaration echoes that `load` prints (shen-lua's file path is `(load FILE)`, which echoes every define), - strip lone `0`/`nil`/`true` load-echo lines that `load` prints as the @@ -271,6 +275,10 @@ def normalize(text, extra_prefixes=()): s = ln.rstrip() stripped = s.strip() if stripped.startswith("run time:"): + # also drop the blank line the banner's leading "\n" produced, so + # the WHOLE cosmetic banner is stripped, not just its text line. + if out and out[-1].strip() == "": + out.pop() continue if stripped.startswith("(fn ") and stripped.endswith(")"): continue diff --git a/cases/divergences.json b/cases/divergences.json index 05ab447..544f32b 100644 --- a/cases/divergences.json +++ b/cases/divergences.json @@ -12,8 +12,7 @@ "mode": "eval", "expr": "(load \"programs/load-echo-probe.shen\")", "expect": "agreement", - "known_divergence": "shen-lua-fasl-load-echo", - "doc": "OPEN divergence. The canonical kernel load (klambda/load.kl -> shen.eval-and-print) PRINTS each toplevel form's evaluated value before returning `loaded`; this is the echo you see from `shen eval -l FILE` and `(load FILE)` at the REPL (NOT `script` mode, which is quiet on every port -- so use eval mode here). shen-rust and shen-go emit it; shen-lua emits it too on a COLD fasl cache or with SHEN_FASL=off, but on a WARM fasl-cache HIT it DROPS the per-form echo (boot.lua ~796; the documented 'a replayed load does not echo per-form values/types' degrade at boot.lua:476). Net effect: `(load file)` stdout is cache-state dependent -- a cross-port divergence (warm shen-lua vs everyone else) AND an intra-port nondeterminism (shen-lua cold != shen-lua warm). Scope: affects toplevel VALUE echo only; `(output ...)`/`(print ...)` side-effects are unaffected and agree everywhere. Minimal repro: eval `(load \"...load-echo-probe.shen\")` (file has \"PROBE\" / (+ 40 2) / (define p -> ok)); canonical prints PROBE, 42, (fn p) then loaded; warm shen-lua prints only loaded. Tracked: pyrex41/shen-lua#40 (fasl replay should re-emit the eval-and-print echo, or exclude the replay path from load's observable-output contract). Flip to a HARD agreement (drop known_divergence) once converged, as hush-file-write was." + "doc": "RESOLVED (was a known divergence): `(load FILE)` now echoes each toplevel form's evaluated value on every impl, cache state notwithstanding. The canonical kernel load (klambda/load.kl -> shen.eval-and-print) PRINTS each toplevel result before returning `loaded` (the echo from `shen eval -l FILE` / `(load FILE)` at the REPL; NOT `script` mode, which is quiet everywhere -- so this case uses eval mode). shen-rust and shen-go always emitted it; shen-lua emitted it on a COLD fasl cache but DROPPED it on a WARM fasl-cache HIT (boot.lua returned `loaded` straight out of fasl_replay without re-running the kernel load), making `(load file)` stdout cache-state dependent -- a cross-port divergence AND an intra-port nondeterminism. Fixed in shen-lua by recording the per-form echo bytes as fasl \"e\" records on the miss and re-pr'ing them on the hit (pyrex41/shen-lua#40, PR #41). The cosmetic `run time:` banner stays dropped on a warm hit by design; the normaliser here strips the whole banner (text line + the blank line its leading newline produces) so that omission is not itself a divergence. Now asserted as a HARD agreement: every available impl must echo `\"PROBE\"` / `42` / `(fn p)` (the last stripped by the normaliser) then `loaded`; any impl that drops the value echo is a real FAIL." } ] }