Record: 11L GradQuant + EMA + Sliding Eval (val_bpb=1.1416)

[albertorkive]

Summary

• val_bpb: 1.1416 (post int8+zstd quantization roundtrip, sliding window eval stride=64, full validation coverage)
• Artifact: 15,059,186 bytes (code: 59,158 bytes + model: 15,000,028 bytes)
• 11 layers, 512 dim, MLP 3x, Muon optimizer, EMA (alpha=0.997, from init)

Key Techniques

• Gradient-guided adaptive quantization: per-tensor int5/int6/int7 bit assignment based on gradient sensitivity (top 30% → int7, middle 40% → int6, bottom 30% → int5)
• EMA from model init (alpha=0.997)
• SmearGate residual mixing + NTK-aware RoPE + XSA (last 4 layers)
• Orthogonal initialization + tied embeddings
• zstd level 22 compression
• Sliding window eval with stride=64, full validation set coverage (~121K windows/GPU)

Run Command

pip install sentencepiece zstandard
python3 data/cached_challenge_fineweb.py
torchrun --nproc_per_node=8 train_gpt.py

All hyperparameters are baked into train_gpt.py as defaults. No env vars needed.

Included Files

• README.md — Architecture and training details
• submission.json — Metadata and metrics
• train_gpt.py — Complete training script (1,309 lines)
• train.log — Full training + evaluation output