Is differential fuzzing the spec interpreter still worth it?

[alexcrichton]

An oss-fuzz bug just got opened which is a timeout in the spec interpreter. In investigating it locally I've discoverd that when my gcc updated a few days ago I can no longer compile the C code in ocaml-boxroot-sys, which ocaml-interop, what we use in Rust, depends on. The latest version of ocaml-interop does at least compile its C code for me, but the new Rust APIs are, at my read, fundamentally incompatible with how we want to invoke OCaml. There's no way that I can see which would initialize the OCaml runtime and a way that can be reused without having to thread it across fuzz executions.

This led me to a broader question of possibly removing the differential fuzzing against the spec interpreter. The points leading me to this conclusion are:

• We haven't had any differential fuzz issues found in years now at this point.
• In the meantime wasmi is on-parity with the spec interpreter in terms of wasm features.
• Differentially fuzzing against N engines, I believe, is more-or-less the same as differential fuzzing against N+1 engines. We'd still be differential-fuzzing against V8, wasmi, and Cranelift vs Winch.
• The spec interpreter integration hasn't been touched in many years. This is still Conrad's old fork of the spec interpreter which fixed some original performance issues, but we haven't updated it in quite awhile. The Rust integration itself is quite old.

Basically it's currently got a fuzz bug, I can't investigate locally due to our Rust deps using now-broken C code, and I also fear we've been keeping it running for quite some time without a whole lot of benefit.

I wanted to canvas others, however. How do others feel about this? (cc @fitzgen and @cfallin)

[cfallin]

Personally, I have on several occasions done a local edit to remove the spec-interpreter feature from fuzz-builds because my OCaml setup is broken. It certainly adds friction.

I think we originally added it because there was a lot of credibility-value to be obtained from "fuzzed against the spec itself" -- and it's the obvious thing to do when the spec is executable. I do think the spec is meaningfully better than "just another engine" in this sense so the N -> N+1 framing isn't quite right IMHO.

That said, all the things you point out (outdated fork, no real issues found) sort of take away from that. The spec itself is also moving a little more slowly these days now that a lot of the semantically-big proposals have been completed (SIMD, GC, exceptions). Stack-switching is still on the table and semantically extremely interesting, but that's also hindered by "old fork".

So I would agree that we get largely the same value from fuzzing against other engines, and I'm in favor of deleting this if it allows us to move forward more easily.

[fitzgen]

Yeah, nothing more to add. Seems like it probably isn't worth it anymore.