Feature: Swept reproducibility matrix + ed25519 verify-before-load

.gitattributes

@@ -0,0 +1 @@
+*.safetensors filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -14,7 +14,28 @@ __pycache__/
 *.pt
 *.pth
 *.jsonl
+*.sig
 eval_manifest.json
 pipeline_manifest.json
 mid_checkpoint.safetensors
 mid_checkpoint.merkle.json
+
+# Python venvs
+.venv/
+
+# ed25519: keep the PUBLIC key (so anyone can verify); never commit the private key
+keys/*.key
+
+# generated experiment outputs and swept artifacts
+results/
+artifacts/
+
+# session/context scratch — never commit
+repo_prompt.md
+
+# downloaded corpora (large); keep only the bundled offline sample
+data/*.txt
+data/*.zip
+data/*.tar.gz
+data/cifar-10-batches-py/
+!data/shakespeare_sample.txt

README.md

@@ -8,6 +8,57 @@ The goal is not just to publish a model, but to publish a model whose training p
 
 ---
 
+## The reproducibility matrix (this build)
+
+Earlier versions hashed a checkpoint and called it verified. The sharper claim this
+build is organised around: **training reproducibility is universally assumed and
+rarely verified — it breaks silently, and this tool surfaces the exact conditions
+under which it fails.** The hook is the *failure*, not the success.
+
+A single parametrized runner sweeps a grid of models × conditions and emits one JSON
+record per cell, deliberately mixing reproducible and broken outcomes:
+
+| | fp32 det-on | fp32 det-off | tf32 | bf16 | cross-GPU |
+|---|---|---|---|---|---|
+| **mlp** (control) | PASS | (verify on GPU) | bits≠fp32 (Ampere) | bits≠fp32 | (verify) |
+| **gpt10m** (attention) | PASS | run-to-run FAIL on GPU | bits≠fp32 | bits≠fp32 | DIFF |
+| **lstm** (cuDNN recurrent) | PASS | FAIL on GPU | bits≠fp32 | bits≠fp32 | DIFF |
+
+Two comparisons are kept deliberately separate, because conflating them is the most
+common reproducibility error:
+
+- **Run-to-run** (`reproducible`, `first_divergence_step`): train the *same* config
+  twice on the *same* hardware — identical bits? Broken by nondeterministic kernels
+  (determinism OFF) and by different GPUs.
+- **Agreement with the fp32 reference** (`vs_fp32`): does tf32/bf16 produce the same
+  bits — or merely the same loss to a tolerance — as fp32? TF32/bf16 are perfectly
+  run-to-run reproducible yet silently disagree with fp32.
+
+**The planted debate.** `verify()` accepts losses within `rel_tol=1e-6` but compares
+parameters by *exact* hash, so a run can pass the loss check and fail the bitwise
+check. Is the right verification bar bitwise identity (strong, brittle, hardware-bound)
+or numerical tolerance (portable, but admits silent precision drift)? The matrix
+supplies evidence both ways; the choice defines what "reproducible training" means.
+
+**Security upgrade.** Checkpoints are now ed25519-signed and the signature is verified
+*before* deserialization (`src/signing.py`). The previous path,
+`torch.load(weights_only=False)` followed by a SHA-256 check, executes arbitrary code
+at unpickle time — before the integrity check ever runs. Signing also makes the
+"cryptographically signed" claim true (a SHA-256 is a checksum, not a signature).
+
+### Quickstart
+
+```bash
+pip install -r requirements.txt
+python demo.py                 # full arc on CPU (smoke config); --full on a GPU pod
+python sweep.py --quick        # the matrix, tiny CPU preset
+cd src && python reproducibility.py   # segmented-replay audit (CLEAN AUDIT PASS + scenarios)
+```
+
+See **RUNBOOK.md** for the exact pod commands that populate the GPU-only cells.
+
+---
+
 ## Why this project exists
 
 Open-weight models are reproducible in principle but not verifiable in practice. You can download the weights, but you cannot prove what data they were trained on, what configuration produced them, or whether they were modified after release. A model ships with a report, and the report has to be trusted.

RUNBOOK.md

@@ -0,0 +1,157 @@
+# RUNBOOK — demo day
+
+Exact commands for the live demo. Everything except the GPU-only failure cells was
+verified on CPU; the GPU cells must be run once on the pod **before** you build
+slides around them (TF32 and bf16 are reliable; determinism-OFF and cross-GPU are
+empirical — verify them first).
+
+The fast path: `python demo.py --full` on the pod runs the whole arc. The phases
+below are for when you want to drive each exhibit yourself.
+
+---
+
+## 0. Pod setup (RunPod PyTorch template, L4 / A40-class)
+
+```bash
+git clone <your repo> && cd OpenVerifiableLLM
+pip install -r requirements.txt          # torch, numpy, safetensors, pynacl, matplotlib, tqdm
+python - <<'PY'
+import torch; print("torch", torch.__version__, "cuda", torch.cuda.is_available(),
+                    torch.cuda.get_device_name(0) if torch.cuda.is_available() else "")
+PY
+python src/signing.py                     # generates keys/ (ed25519) + self-check
+```
+
+The first text run auto-downloads tinyshakespeare (~1 MB). For the bigger corpora:
+`--dataset enwik8` (downloads ~36 MB) or `--dataset wikitext` (needs `pip install datasets`).
+
+---
+
+## 1. Headline: the same GPU is bitwise reproducible (and the audit passes)
+
+```bash
+cd src
+python gpu_reproducibility_test.py        # fresh-vs-fresh: identical bits run-to-run
+python reproducibility.py                 # CLEAN AUDIT PASS + the 5 scenarios
+cd ..
+```
+
+What to say: "With a pinned cuBLAS workspace and deterministic algorithms, training
+this 11M-param model twice on this GPU gives **the same bits** — and the segmented
+replay audit confirms a checkpoint can be resumed bit-for-bit." Scenario 5 shows a
+tampered checkpoint **rejected before deserialization** (the security fix).
+
+CPU smoke (proves plumbing without waiting on a GPU):
+```bash
+cd src && OVL_TOTAL_STEPS=10 OVL_CHECKPOINT_STEP=5 OVL_BATCH_SIZE=4 OVL_BLOCK_SIZE=64 \
+    python reproducibility.py ; cd ..
+```
+
+---
+
+## 2. The matrix (the spread of PASS/FAIL)
+
+```bash
+python sweep.py --device cuda --track-divergence
+# add the scale + modality rows:
+python sweep.py --device cuda --stretch --track-divergence
+```
+
+Writes `results/results.jsonl` and prints the grid. Expect: fp32+det-on reproducible;
+bf16 + tf32 bit-differ from the fp32 reference; determinism-OFF breaks run-to-run on
+GPU (first-divergence step populated).
+
+---
+
+## 3. Single cells for slides
+
+```bash
+# control
+python run_experiment.py --model gpt10m --precision fp32 --deterministic on  --device cuda
+# the silent killer (Ampere default): loss barely moves, bits change
+python run_experiment.py --model gpt10m --precision tf32 --deterministic on  --device cuda
+# half precision
+python run_experiment.py --model gpt10m --precision bf16 --deterministic on  --device cuda
+# determinism OFF: run-to-run divergence
+python run_experiment.py --model gpt10m --precision fp32 --deterministic off --device cuda --track-divergence
+```
+
+---
+
+## 4. Cross-GPU column (verify before relying on it)
+
+`use_deterministic_algorithms` holds **per hardware stack**, not across architectures.
+
+```bash
+# On GPU type A (e.g. L4):
+python sweep.py --device cuda --out results/results_L4.jsonl
+# Stop the pod, switch the GPU type to A40, restart, then:
+python sweep.py --device cuda --out results/results_A40.jsonl
+# Compare param hashes per cell:
+python - <<'PY'
+import json
+load=lambda p:{(o["model"],o.get("condition")):o["param_sha256"] for o in map(json.loads,open(p))}
+a,b=load("results/results_L4.jsonl"),load("results/results_A40.jsonl")
+for k in sorted(a):
+    print(f"{k[0]:>7} {str(k[1]):<12} {'SAME' if a[k]==b.get(k) else 'DIFF'}")
+PY
+```
+
+Then fill the demo's CROSS-GPU column:
+```bash
+python demo.py --full --device cuda --cross-gpu-results results/results_A40.jsonl
+```
+
+---
+
+## 5. T7 — the divergence-accumulation plot
+
+```bash
+# determinism OFF, hashing params every step in two twin runs:
+python run_experiment.py --model gpt10m --precision fp32 --deterministic off \
+    --device cuda --track-divergence --out results/divergence.jsonl
+python src/plot_divergence.py results/divergence.jsonl   # -> results/divergence.png
+```
+
+The plot marks the step where two "identical" runs first differ — the most novel
+artifact in the deck.
+
+---
+
+## 6. Stretch (only if green)
+
+```bash
+python run_experiment.py --model gpt50m --precision fp32 --deterministic on --device cuda   # scale axis
+python run_experiment.py --model cnn --dataset cifar --precision fp32 --deterministic off --device cuda  # conv nondeterminism
+# DDP (>=2 GPUs): all-reduce ordering vs bitwise repro
+torchrun --nproc_per_node=2 src/ddp_repro.py
+```
+
+A negative DDP result (cannot get bitwise-identical across ranks) is still the real
+open problem — present it as such.
+
+---
+
+## 7. Dry run + fallback
+
+```bash
+time python demo.py --full --device cuda          # one timed full pass the night before
+```
+
+Record a clean `demo.py --full` screen-capture as a fallback in case the pod is flaky
+on stage. Keep `results/results.jsonl` from a good run as a static backup for the grid.
+
+---
+
+## Troubleshooting
+
+- **"deterministic algorithm not available"** on an exotic op: the sweep already runs
+  with `warn_only=True`; the audit's control uses strict mode. If the audit crashes on
+  a specific op, run that cell through `sweep.py` (warn_only) instead.
+- **CUBLAS_WORKSPACE_CONFIG**: set automatically on import of `device`. If you see a
+  cuBLAS determinism error, confirm nothing imported torch and touched CUDA before
+  `device`/`main`.
+- **tf32 shows SAME on CPU**: expected — TF32 is an Ampere+ GPU feature; the divergence
+  only appears on `--device cuda`.
+- **Dataset download blocked**: `--dataset shakespeare` falls back to the bundled
+  `data/shakespeare_sample.txt`; determinism results stay valid.

data/shakespeare_sample.txt

@@ -0,0 +1,131 @@
+From fairest creatures we desire increase,
+That thereby beauty's rose might never die,
+But as the riper should by time decease,
+His tender heir might bear his memory:
+But thou, contracted to thine own bright eyes,
+Feed'st thy light's flame with self-substantial fuel,
+Making a famine where abundance lies,
+Thy self thy foe, to thy sweet self too cruel:
+Thou that art now the world's fresh ornament,
+And only herald to the gaudy spring,
+Within thine own bud buriest thy content,
+And tender churl mak'st waste in niggarding:
+  Pity the world, or else this glutton be,
+  To eat the world's due, by the grave and thee.
+
+Shall I compare thee to a summer's day?
+Thou art more lovely and more temperate:
+Rough winds do shake the darling buds of May,
+And summer's lease hath all too short a date:
+Sometime too hot the eye of heaven shines,
+And often is his gold complexion dimm'd;
+And every fair from fair sometime declines,
+By chance or nature's changing course untrimm'd;
+But thy eternal summer shall not fade
+Nor lose possession of that fair thou ow'st;
+Nor shall Death brag thou wander'st in his shade,
+When in eternal lines to time thou grow'st:
+  So long as men can breathe or eyes can see,
+  So long lives this, and this gives life to thee.
+
+When, in disgrace with fortune and men's eyes,
+I all alone beweep my outcast state,
+And trouble deaf heaven with my bootless cries,
+And look upon myself and curse my fate,
+Wishing me like to one more rich in hope,
+Featur'd like him, like him with friends possess'd,
+Desiring this man's art and that man's scope,
+With what I most enjoy contented least;
+Yet in these thoughts myself almost despising,
+Haply I think on thee, and then my state,
+Like to the lark at break of day arising
+From sullen earth, sings hymns at heaven's gate;
+  For thy sweet love remember'd such wealth brings
+  That then I scorn to change my state with kings.
+
+Let me not to the marriage of true minds
+Admit impediments. Love is not love
+Which alters when it alteration finds,
+Or bends with the remover to remove:
+O no! it is an ever-fixed mark
+That looks on tempests and is never shaken;
+It is the star to every wand'ring bark,
+Whose worth's unknown, although his height be taken.
+Love's not Time's fool, though rosy lips and cheeks
+Within his bending sickle's compass come;
+Love alters not with his brief hours and weeks,
+But bears it out even to the edge of doom.
+  If this be error and upon me prov'd,
+  I never writ, nor no man ever lov'd.
+
+To be, or not to be, that is the question:
+Whether 'tis nobler in the mind to suffer
+The slings and arrows of outrageous fortune,
+Or to take arms against a sea of troubles
+And by opposing end them. To die-to sleep,
+No more; and by a sleep to say we end
+The heart-ache and the thousand natural shocks
+That flesh is heir to: 'tis a consummation
+Devoutly to be wish'd. To die, to sleep;
+To sleep, perchance to dream-ay, there's the rub:
+For in that sleep of death what dreams may come,
+When we have shuffled off this mortal coil,
+Must give us pause-there's the respect
+That makes calamity of so long life.
+
+All the world's a stage,
+And all the men and women merely players;
+They have their exits and their entrances,
+And one man in his time plays many parts,
+His acts being seven ages. At first, the infant,
+Mewling and puking in the nurse's arms.
+Then the whining schoolboy, with his satchel
+And shining morning face, creeping like snail
+Unwillingly to school. And then the lover,
+Sighing like furnace, with a woeful ballad
+Made to his mistress' eyebrow. Then a soldier,
+Full of strange oaths and bearded like the pard,
+Jealous in honour, sudden and quick in quarrel,
+Seeking the bubble reputation
+Even in the cannon's mouth.
+
+Tomorrow, and tomorrow, and tomorrow,
+Creeps in this petty pace from day to day,
+To the last syllable of recorded time;
+And all our yesterdays have lighted fools
+The way to dusty death. Out, out, brief candle!
+Life's but a walking shadow, a poor player,
+That struts and frets his hour upon the stage,
+And then is heard no more. It is a tale
+Told by an idiot, full of sound and fury,
+Signifying nothing.
+
+Now is the winter of our discontent
+Made glorious summer by this sun of York;
+And all the clouds that lour'd upon our house
+In the deep bosom of the ocean buried.
+Now are our brows bound with victorious wreaths;
+Our bruised arms hung up for monuments;
+Our stern alarums chang'd to merry meetings,
+Our dreadful marches to delightful measures.
+
+Friends, Romans, countrymen, lend me your ears;
+I come to bury Caesar, not to praise him.
+The evil that men do lives after them;
+The good is oft interred with their bones;
+So let it be with Caesar. The noble Brutus
+Hath told you Caesar was ambitious:
+If it were so, it was a grievous fault,
+And grievously hath Caesar answer'd it.
+
+O Romeo, Romeo! wherefore art thou Romeo?
+Deny thy father and refuse thy name;
+Or, if thou wilt not, be but sworn my love,
+And I'll no longer be a Capulet.
+'Tis but thy name that is my enemy;
+Thou art thyself, though not a Montague.
+What's Montague? it is nor hand, nor foot,
+Nor arm, nor face, nor any other part
+Belonging to a man. O, be some other name!
+What's in a name? that which we call a rose
+By any other name would smell as sweet.