Multi-threaded multi-path Pathfinder broken with recent Transducers versions

[sethaxen]

Transducers v0.4.76 has caused our tests to fail (see first failure here).

Some not-necessarily-relevant notes:

• In addition to the merged PRs in those release notes, Add fast pathway for copy, collect, tcollect, and tcopy for size-stable operations JuliaFolds/Transducers.jl#553 was also merged, which supposedly resolved foldxd fails over Iterators.Product JuliaFolds/Transducers.jl#552
• We currently use a deprecated method (from https://github.com/mlcolab/Pathfinder.jl/actions/runs/4920822614/jobs/8790013138#step:6:775):

┌ Warning: `reduce(rf, itr::Foldable; kw...)` is deprecated, use `foldxt(rf, itr; kw...)` instead.
│   caller = multipathfinder(optim_fun::OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{LogDensityFunctionWithGradHess{var"#logp#44"{FullNormal}, var"#∇f#40"{var"#logp#44"{FullNormal}}, var"#Hf#41"{var"#logp#44"{FullNormal}}}}, Pathfinder.var"#grad#6"{LogDensityFunctionWithGradHess{var"#logp#44"{FullNormal}, var"#∇f#40"{var"#logp#44"{FullNormal}}, var"#Hf#41"{var"#logp#44"{FullNormal}}}}, Pathfinder.var"#hess#7"{LogDensityFunctionWithGradHess{var"#logp#44"{FullNormal}, var"#∇f#40"{var"#logp#44"{FullNormal}}, var"#Hf#41"{var"#logp#44"{FullNormal}}}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ndraws::Int64; init::Nothing, input::LogDensityFunctionWithGradHess{var"#logp#44"{FullNormal}, var"#∇f#40"{var"#logp#44"{FullNormal}}, var"#Hf#41"{var"#logp#44"{FullNormal}}}, nruns::Int64, ndraws_elbo::Int64, ndraws_per_run::Int64, rng::MersenneTwister, history_length::Int64, optimizer::LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, executor::SequentialEx{NamedTuple{(), Tuple{}}}, executor_per_run::SequentialEx{NamedTuple{(), Tuple{}}}, importance::Bool, kwargs::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}) at multipath.jl:182
└ @ Pathfinder ~/work/Pathfinder.jl/Pathfinder.jl/src/multipath.jl:182
rng = TaskLocalRNG(): Error During Test at /home/runner/work/Pathfinder.jl/Pathfinder.jl/test/multipath.jl:31

[sethaxen]

Here's an MWE for this error:

using LogDensityProblems, Pathfinder, Transducers

struct StdNormalProblem
    n::Int
end
LogDensityProblems.capabilities(::Type{StdNormalProblem}) = LogDensityProblems.LogDensityOrder{1}()
LogDensityProblems.dimension(prob::StdNormalProblem) = prob.n
LogDensityProblems.logdensity(prob::StdNormalProblem, x) = -sum(abs2, x) / 2
LogDensityProblems.logdensity_and_gradient(prob::StdNormalProblem, x) = (-sum(abs2, x) / 2, -x)

ndraws = 100
nruns = 4
prob = StdNormalProblem(2)
multipathfinder(prob, ndraws; nruns)  # fine
multipathfinder(prob, ndraws; nruns, executor=ThreadedEx())  # errors, see below

error

ERROR: MethodError: no method matching iterate(::Transducers.ProgressLoggingFoldable{Vector{Nothing}})

Closest candidates are:
  iterate(::Union{LinRange, StepRangeLen})
   @ Base range.jl:880
  iterate(::Union{LinRange, StepRangeLen}, ::Integer)
   @ Base range.jl:880
  iterate(::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}}
   @ Base dict.jl:698
  ...

Stacktrace:
  [1] iterate(::Base.Iterators.PartitionIterator{Transducers.ProgressLoggingFoldable{Vector{Nothing}}})
    @ Base.Iterators ./iterators.jl:1319
  [2] copyto!(dest::Vector{Vector{Nothing}}, src::Base.Iterators.PartitionIterator{Transducers.ProgressLoggingFoldable{Vector{Nothing}}})
    @ Base ./abstractarray.jl:946
  [3] _collect(cont::UnitRange{Int64}, itr::Base.Iterators.PartitionIterator{Transducers.ProgressLoggingFoldable{Vector{Nothing}}}, #unused#::Base.HasEltype, isz::Base.HasLength)
    @ Base ./array.jl:713
  [4] collect(itr::Base.Iterators.PartitionIterator{Transducers.ProgressLoggingFoldable{Vector{Nothing}}})
    @ Base ./array.jl:707
  [5] split_into_chunks(coll::Transducers.ProgressLoggingFoldable{Vector{Nothing}}, sz::Int64)
    @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:364
  [6] _tcopy(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, ::Type{Vector}, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}, ::Transducers.SizeStable, ::Base.HasShape{1}; basesize::Int64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:358
  [7] _tcopy(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, ::Type{Vector}, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}, ::Transducers.SizeStable, ::Base.HasShape{1})
    @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:356
  [8] tcopy(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, T::Type, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:354
  [9] tcopy(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, T::Type, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}})
    @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:354
 [10] tcollect(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:434
 [11] tcollect(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}})
    @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:434
 [12] collect(itr::Transducers.Eduction{Transducers.Reduction{Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, Transducers.BottomRF{Completing{typeof(push!!)}}}, Transducers.ProgressLoggingFoldable{Vector{Nothing}}}, ex::ThreadedEx{NamedTuple{(), Tuple{}}})
    @ Folds.Implementations ~/.julia/packages/Folds/ZayPF/src/collect.jl:12
 [13] collect(itr::Transducers.Eduction{Transducers.Reduction{Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, Transducers.BottomRF{Completing{typeof(push!!)}}}, Transducers.ProgressLoggingFoldable{Vector{Nothing}}}, ex::PreferParallel{NamedTuple{(), Tuple{}}})
    @ Folds.Implementations ~/.julia/packages/Folds/ZayPF/src/collect.jl:9
 [14] multipathfinder(optim_fun::SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ndraws::Int64; init::Nothing, input::StdNormalProblem, nruns::Int64, ndraws_elbo::Int64, ndraws_per_run::Int64, rng::Random._GLOBAL_RNG, history_length::Int64, optimizer::Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, executor::PreferParallel{NamedTuple{(), Tuple{}}}, executor_per_run::SequentialEx{NamedTuple{(), Tuple{}}}, importance::Bool, kwargs::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}})
    @ Pathfinder ~/.julia/packages/Pathfinder/BUjTN/src/multipath.jl:179
 [15] multipathfinder(ℓ::StdNormalProblem, ndraws::Int64; input::StdNormalProblem, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:nruns, :executor), Tuple{Int64, PreferParallel{NamedTuple{(), Tuple{}}}}}})
    @ Pathfinder ~/.julia/packages/Pathfinder/BUjTN/src/multipath.jl:130
 [16] top-level scope
    @ REPL[13]:1

@marcobonici is this the error you observed when using Pathfinder with executor=PreferParallel()?

[sethaxen]

This new failure in Transducers might be related: JuliaFolds/Transducers.jl#557

[marcobonici]

Hi @sethaxen , yes, this is the same error I found. In order to deal with it I just downgraded to the previous version of Transducers.jl, but I can understand that from a package mantainer point of view this is not satisfactory 😅

[sethaxen]

The new issue to follow is JuliaFolds2/Transducers.jl#10