Test against Enzyme

[wsmoses]

Let's see what fails.

Testing things is useful to make sure things continue to work and don't accidentally break in the future :)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.51%. Comparing base (f3bd120) to head (6c32c7b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2636   +/-   ##
=======================================
  Coverage   86.51%   86.51%           
=======================================
  Files          20       20           
  Lines        1261     1261           
=======================================
  Hits         1091     1091           
  Misses        170      170           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coveralls]

Pull Request Test Coverage Report for Build 17703888586

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 83.719%

---

Totals
Change from base Build 17557004860:	0.0%
Covered Lines:	1193
Relevant Lines:	1425

---

💛 - Coveralls

[wsmoses]

hm good progress but some variation hits a segfault (though notably not within Enzyme code funny enough).

@penelopeysm and/or @mhauru would you be able to extract a reproducer for me, and I can make sure it gets fixed

[penelopeysm]

Hi @wsmoses I didn't look at all of them individually but I don't see which CI run is failing due to segfaults.

• Most of them are failing because of the coverage provider's servers being wonky.
• There are some known failures on 1.12 which are already on main
• ad.jl is mainly failing because you need to broaden the type of eltypes_by_adtype on line 35 of test/ad.jl

- eltypes_by_adtype = Dict(
+ eltypes_by_adtype = Dict{Type,Tuple}(

It's likely that the adtype needs function_annotation=Enzyme.Const as well. I have these changes locally and would be happy to push these changes myself but I don't have the permissions to push to this branch (not sure if there is a checkbox for you to allow maintainers to push changes).

It's of course possible that making these changes will reveal new errors, but we can deal with that if it happens then.

More broadly speaking though testing on Turing is not very diagnostic IMO. Of course happy to add tests but my point is that the vast majority of things will be caught in DynamicPPL and if AD tests pass there I would say with ≥95% confidence that it will pass here.

Regarding DynamicPPL, everything passes fine on the current release of v0.36 but Enzyme has been failing on v0.37. I already posted an MWE at EnzymeAD/Enzyme.jl#2429 (see TuringLang/DynamicPPL.jl#985 if you'd like a CI log) so IMO from our side that's the highest priority thing that we'd like to get fixed, although I've not chased up on it since I'm not sure when 0.37 will actually be released.

[penelopeysm]

There's a bunch of

Enzyme compilation failed due to illegal type analysis.
   This usually indicates the use of a Union type, which is not fully supported with Enzyme.API.strictAliasing set to true [the default].
   Ideally, remove the union (which will also make your code faster), or try setting Enzyme.API.strictAliasing!(false) before any autodiff call.
   To toggle more information for debugging (needed for bug reports), set Enzyme.Compiler.VERBOSE_ERRORS[] = true (default false)
   Failure within method: MethodInstance for push!(::DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, typeof(identity)}, Int64}, Vector{Distributions.DiagNormal}, Vector{AbstractPPL.VarName{:m, typeof(identity)}}, Vector{Float64}}, ::AbstractPPL.VarName{:m, typeof(identity)}, ::Vector{Float64}, ::Distributions.DiagNormal, ::Int64)
  Hint: catch this exception as `err` and call `code_typed(err)` to inspect the errornous code.
  If you have Cthulu.jl loaded you can also use `code_typed(err; interactive = true)` to interactively introspect the code.
  Caused by:
  Stacktrace:
   [1] <
     @ ./int.jl:83
   [2] _growend!
     @ ./array.jl:1115
   [3] push!
     @ ./array.jl:1261
   [4] push!
     @ ~/.julia/packages/DynamicPPL/nxcz4/src/varinfo.jl:1809

There are pretty huge changes to DynamicPPL coming up that will involve this code path so I will wait for the dust around that to settle first.

On a more general note I don't think it's feasible to simply test everything with Enzyme forward + reverse because the compilation just makes CI take a really long time which is kind of unfair. I think it would be a sensible option to define a smaller set of Enzyme AD tests and run them separately (similar to what we do on DynamicPPL).

[wsmoses]

tangentially, now that it seems that all of the dynamicppl thigns are settled, can we try to narrow down the remaining failures?

[yebai]

For future reference: The Turing.jl team cannot help with Enzyme bugs. Please report issues directly to the Enzyme repository. As a small research team, we don't have the capacity to help minimize or debug Enzyme-related issues.