Skip to content

Add generalized semiring reductions to einsum VM and JIT - #132

Open
imlvts wants to merge 5 commits into
trueagi-io:mainfrom
imlvts:einsum-reduce
Open

Add generalized semiring reductions to einsum VM and JIT#132
imlvts wants to merge 5 commits into
trueagi-io:mainfrom
imlvts:einsum-reduce

Conversation

@imlvts

@imlvts imlvts commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Generalize the einsum contraction to an arbitrary commutative semiring: a Reduce operator {Sum, Prod, Max, Min} folding contributions into the output, over a Combine operator {Mul, Add, Min, Max} merging the input operands — so max-product (A[i,j,k] = max_q B[j]*C[i,k,q]*D[k,q]) and min-plus (tropical / shortest-path) contractions run through the same machinery as plain sum-product einsum. The design is modelled after the Julia library Tullio.jl (noted as a footnote in the module docs).

  • New Scalar element trait (tensor.rs) carrying the per-operator identity table: Sum→0, Prod→1, Max→-inf/T::MIN, Min→+inf/T::MAX. Empty reductions yield the identity.
  • VM: compile_reduce / einsum_reduce / einsum_reduce_dyn. The einsum_reduce wrappers pre-fill outputs with the reduce identity (overwrite semantics); compile_reduce + Program::exec keeps accumulate semantics.
  • Sparse row iteration (structural-zero skipping) is only sound when a missing operand annihilates its contribution to the reduce identity — precisely (Sum, Mul). Other op pairs iterate sparse inputs densely in the VM; the JIT rejects them with Unsupported.
  • JIT: EinsumF32Jit::compile_reduce / einsum_jit_reduce, swapping fmax/fmin/fmul/fadd into the existing register-accumulator and RMW paths. (Sum, Mul) emits IR identical to before.
  • Tests: semiring unit tests (incl. integer MIN-identity, empty ranges, AccFlush RMW edge, sparse-with-max dense fallback) plus a 481k-case differential sweep (tests/reduce_sweep.rs) comparing VM and JIT against a naive reference over all specs on {a,b,c}, 7 op pairs, all sparse masks — bit-exact. Existing 19.5M-case einsum_sweep passes unchanged.

VM (interpreted) einsum regressed ~5–15%:

Benchmark (VM einsum, base 5464713 → head f8b6fe0) base head Δ
Dense matmul 16×16, einsum_homogenous (concrete) 65.6 µs 69.5 µs +6.0%
Dense matmul 64×64, einsum_homogenous (concrete) 3.72 ms 3.97 ms +6.8%
Dense matmul 128×128, einsum_homogenous (concrete) 29.3 ms 31.3 ms +6.7%
Dense matmul 256×256, einsum_homogenous (concrete) 232.5 ms 247.5 ms +6.5%
Dense matmul 16×16, einsum (&dyn) 77.4 µs 86.7 µs +12.1%
Dense matmul 64×64, einsum (&dyn) 4.46 ms 5.05 ms +13.3%
Dense matmul 128×128, einsum (&dyn) 35.1 ms 41.0 ms +17.0%
Dense matmul 256×256, einsum (&dyn) 279.0 ms 317.2 ms +13.7%
Dense matmul 128×128, einsum_homogenous (Tensor) 36.3 ms 38.5 ms +6.2%
Dense matmul 256×256, einsum_homogenous (Tensor) 276.8 ms 303.0 ms +9.5%
CSR×CSR side=5, einsum_homogenous (Csr) 47.7 µs 52.3 µs +9.5%
CSR×CSR side=10, einsum_homogenous (Csr) 518.2 µs 561.0 µs +8.3%
CSR×CSR side=20, einsum_homogenous (Csr) 56.0 ms 57.1 ms +2.0%
CSR×Dense 1000×1000 × 1000×8, einsum (&dyn) 576.9 µs 636.5 µs +10.3%
CSR×Dense 8000×8000 × 8000×16, einsum (&dyn) 7.93 ms 8.84 ms +11.6%
CSR×Dense 8000×8000 × 8000×64, einsum (&dyn) 26.1 ms 29.5 ms +13.1%
JIT matmul 128×128 (unaffected) 2.26 ms 2.30 ms +1.6%
JIT matmul 256×256 (unaffected) 16.35 ms 16.48 ms +0.8%

Generalize the einsum contraction to an arbitrary commutative semiring:
a Reduce operator {Sum, Prod, Max, Min} folding contributions into the
output, over a Combine operator {Mul, Add, Min, Max} merging the input
operands — so max-product (`A[i,j,k] = max_q B[j]*C[i,k,q]*D[k,q]`) and
min-plus (tropical / shortest-path) contractions run through the same
machinery as plain sum-product einsum. The design is modelled after the
Julia library Tullio.jl (noted as a footnote in the module docs).

- New `Scalar` element trait (tensor.rs) carrying the per-operator
  identity table: Sum→0, Prod→1, Max→-inf/T::MIN, Min→+inf/T::MAX.
  Empty reductions yield the identity.
- VM: `compile_reduce` / `einsum_reduce` / `einsum_reduce_dyn`. The
  einsum_reduce wrappers pre-fill outputs with the reduce identity
  (overwrite semantics); compile_reduce + Program::exec keeps
  accumulate semantics.
- Sparse row iteration (structural-zero skipping) is only sound when a
  missing operand annihilates its contribution to the reduce identity —
  precisely (Sum, Mul). Other op pairs iterate sparse inputs densely in
  the VM; the JIT rejects them with Unsupported.
- JIT: `EinsumF32Jit::compile_reduce` / `einsum_jit_reduce`, swapping
  fmax/fmin/fmul/fadd into the existing register-accumulator and RMW
  paths. (Sum, Mul) emits IR identical to before.
- Tests: semiring unit tests (incl. integer MIN-identity, empty ranges,
  AccFlush RMW edge, sparse-with-max dense fallback) plus a 481k-case
  differential sweep (tests/reduce_sweep.rs) comparing VM and JIT
  against a naive reference over all specs on {a,b,c}, 7 op pairs, all
  sparse masks — bit-exact. Existing 19.5M-case einsum_sweep passes
  unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
imlvts and others added 4 commits July 14, 2026 21:48
A small decoder-only transformer whose every contraction runs through the
einsum VM, cross-checked against a NumPy reference and a PyTorch trainer that
all share one set of einsum specs.

- train.py: PyTorch nanoGPT-style trainer; defines the arch with torch.einsum
  (same specs as the decode step), trains char-level on an embedded corpus,
  exports weights.bin/config.txt/vocab.bin/prompt.txt.
- gpt2_reference.py: NumPy reference forward (np.einsum, identical specs);
  greedy-decodes and dumps ref_logits.bin.
- main.rs: loads the trained weights at runtime (dims from config), runs the
  KV-cached decode through einsum_homogenous, and reports max-abs logit diff
  and argmax agreement vs the reference. Falls back to random-weight smoke
  test when weights/ is absent.
- README.md documents the architecture, specs, and run steps.

Rust vs NumPy: 24/24 argmax agreement, max abs logit diff 1.7e-5 (float
summation order). No library changes needed — the whole transformer expresses
in the existing einsum plus trivial elementwise helpers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a unary `map(x, f)` helper and rewrite relu, scale, and rmsnorm's
scaling step to use it instead of reading Dense.data directly. add (binary
zip) and softmax_last (per-row) aren't single-function maps and are left as-is.
No behavior change — the example still matches the NumPy reference (24/24
argmax, 1.7e-5 max logit diff).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`cargo run --release --example gpt2 -- bench` times full cold-cache greedy
decodes (best of several batches) and reports per-token latency and tokens/s.
Extracts the decode loop into a reusable run_decode(); the normal compare path
is unchanged and still matches the NumPy reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The intro claimed the whole forward pass "expresses entirely in einsum".
RMSNorm, ReLU, softmax, residual adds, and the attention scale are plain
loops, not einsum — reword to scope the claim to the linear-algebra
contractions, matching the main.rs doc comment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MesTTo

MesTTo commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

the VM regression looks like per-element dispatch: Reduce::apply and Combine::apply match on the enum, and the inner loop reads self.combine/self.reduce and calls them per element. the JIT resolves the op at codegen so it's fine, the VM pays the match every element.

tried lifting (Reduce, Combine) to type params and branching once in exec, +133/-18 in einsum.rs only. by instruction count the delta is -6.84%, both arms at 0.00% variance:

jit_bench, same fixed workload, instructions retired:
  this PR:  ~2.030 trillion
  +patch:   ~1.891 trillion   (-6.84%)

jit_bench.rs is byte-identical across both builds so the whole delta is the einsum change, and the JIT path is in that total too so the VM-only drop is a bit bigger. correctness holds with it applied: the linalg suite, your 481,278-case reduce sweep, and the 19,465,300-case exhaustive sweep all pass.

two caveats: i only measured (Sum, Mul), jit_bench doesn't run min-plus or max-product so i haven't checked those. and the extra instantiations cost size, ~+4.8% here and more on the rlib.

patch's here if it's useful: https://gist.github.com/MesTTo/0d470c97bfecfe503f79e026a537abe5

@MesTTo

MesTTo commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

accidental comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants