Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ version = "0.8.13"

[deps]
ADTypes = "47edcb42-4c32-4615-8424-f2b9edc5f35b"
DifferentiationInterface = "a0c0ee7d-e4b9-4e03-894e-1c5f64a51d63"
ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
NLPModels = "a4795742-8479-5a88-8948-cc11e1c8c1a6"
Expand All @@ -19,10 +20,11 @@ Enzyme = "7da242da-08ed-463a-9acd-ee780be4f1d9"
ADNLPModelsEnzymeExt = "Enzyme"

[compat]
ADTypes = "1.2.1"
ForwardDiff = "1"
NLPModels = "0.21.5"
ReverseDiff = "1"
ADTypes = "1.21.0"
DifferentiationInterface = "0.7.16"
ForwardDiff = "1.3.2"
NLPModels = "0.21.8"
ReverseDiff = "1.16.2"
SparseConnectivityTracer = "1"
SparseMatrixColorings = "0.4.21"
Enzyme = "0.13.129"
Expand Down
2 changes: 1 addition & 1 deletion docs/Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Documenter = "1.0"
ManualNLPModels = "0.2"
NLPModels = "0.21.5"
NLPModelsJuMP = "0.13"
OptimizationProblems = "0.8"
OptimizationProblems = "0.9"
Percival = "0.7"
Plots = "1"
SolverBenchmark = "0.6"
Expand Down
20 changes: 10 additions & 10 deletions docs/src/backend.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
# How to switch backend in ADNLPModels

`ADNLPModels` allows the use of different backends to compute the derivatives required within NLPModel API.
It uses `ForwardDiff.jl`, `ReverseDiff.jl`, and more via optional depencies.
It uses `ForwardDiff.jl`, `ReverseDiff.jl`, and more via optional dependencies.

The backend information is in a structure [`ADNLPModels.ADModelBackend`](@ref) in the attribute `adbackend` of a `ADNLPModel`, it can also be accessed with [`get_adbackend`](@ref).

The functions used internally to define the NLPModel API and the possible backends are defined in the following table:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just switch fully to the ADTypes specification? You're gonna run into trouble translating symbols into AbstractADType objects

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the symbols don't allow you to set parameters like

  • the number of chunks in ForwardDiff
  • the tape compilation in ReverseDiff
  • aspects of the mode in Enzyme


| Functions | FowardDiff backends | ReverseDiff backends | Enzyme backend | Sparse backend |
| --------- | ------------------- | -------------------- | -------------- | -------------- |
| `gradient` and `gradient!` | `ForwardDiffADGradient`/`GenericForwardDiffADGradient` | `ReverseDiffADGradient`/`GenericReverseDiffADGradient` | `EnzymeReverseADGradient` | -- |
| `jacobian` | `ForwardDiffADJacobian` | `ReverseDiffADJacobian` | `SparseEnzymeADJacobian` | `SparseADJacobian` |
| `hessian` | `ForwardDiffADHessian` | `ReverseDiffADHessian` | `SparseEnzymeADHessian` | `SparseADHessian`/`SparseReverseADHessian` |
| `Jprod` | `ForwardDiffADJprod`/`GenericForwardDiffADJprod` | `ReverseDiffADJprod`/`GenericReverseDiffADJprod` | `EnzymeReverseADJprod` | -- |
| `Jtprod` | `ForwardDiffADJtprod`/`GenericForwardDiffADJtprod` | `ReverseDiffADJtprod`/`GenericReverseDiffADJtprod` | `EnzymeReverseADJtprod` | -- |
| `Hvprod` | `ForwardDiffADHvprod`/`GenericForwardDiffADHvprod` | `ReverseDiffADHvprod`/`GenericReverseDiffADHvprod` | `EnzymeReverseADHvprod` | -- |
| `directional_second_derivative` | `ForwardDiffADGHjvprod` | -- | -- | -- |
| Functions | FowardDiff backends | ReverseDiff backends | Enzyme backend | Sparse backend | DI backend |
| --------- | ------------------- | -------------------- | -------------- | -------------- | -------------- |
| `gradient` and `gradient!` | `ForwardDiffADGradient`/`GenericForwardDiffADGradient` | `ReverseDiffADGradient`/`GenericReverseDiffADGradient` | `EnzymeReverseADGradient` | -- | `DIADGradient` |
| `jacobian` | `ForwardDiffADJacobian` | `ReverseDiffADJacobian` | `SparseEnzymeADJacobian` | `SparseADJacobian` | `DIADJacobian` / `SparseDIJacobian` |
| `hessian` | `ForwardDiffADHessian` | `ReverseDiffADHessian` | `SparseEnzymeADHessian` | `SparseADHessian`/`SparseReverseADHessian` | `DIADHessian` / `SparseDIADHessian` |
| `Jprod` | `ForwardDiffADJprod`/`GenericForwardDiffADJprod` | `ReverseDiffADJprod`/`GenericReverseDiffADJprod` | `EnzymeReverseADJprod` | -- | `DIADJprod` |
| `Jtprod` | `ForwardDiffADJtprod`/`GenericForwardDiffADJtprod` | `ReverseDiffADJtprod`/`GenericReverseDiffADJtprod` | `EnzymeReverseADJtprod` | -- | `DIADJtprod` |
| `Hvprod` | `ForwardDiffADHvprod`/`GenericForwardDiffADHvprod` | `ReverseDiffADHvprod`/`GenericReverseDiffADHvprod` | `EnzymeReverseADHvprod` | -- | `DIADHvprod` |
| `directional_second_derivative` | `ForwardDiffADGHjvprod` | -- | -- | -- | -- |

The functions `hess_structure!`, `hess_coord!`, `jac_structure!` and `jac_coord!` defined in `ad.jl` are generic to all the backends for now.

Expand Down
2 changes: 1 addition & 1 deletion docs/src/predefined.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ ADNLPModels.predefined_backend[:optimized]

The backend `:generic` focuses on backend that make no assumptions on the element type, see [Creating an ADNLPModels backend that supports multiple precisions](https://jso.dev/tutorials/generic-adnlpmodels/).

It is possible to use these pre-defined backends using the keyword argument `backend` when instantiating the model.
It is possible to use these pre-defined backends by using the keyword argument `backend` when instantiating the model.

```@example ex1
nlp = ADNLPModel!(f, x0, lvar, uvar, c!, lcon, ucon, backend = :optimized)
Expand Down
4 changes: 3 additions & 1 deletion src/ADNLPModels.jl
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ module ADNLPModels
using LinearAlgebra, SparseArrays

# external
using ADTypes: ADTypes, AbstractColoringAlgorithm, AbstractSparsityDetector
import DifferentiationInterface
using ADTypes: ADTypes, AbstractADType, AbstractColoringAlgorithm, AbstractSparsityDetector, AutoForwardDiff, AutoReverseDiff
using SparseConnectivityTracer: TracerSparsityDetector
using SparseMatrixColorings
using ForwardDiff, ReverseDiff
Expand All @@ -27,6 +28,7 @@ include("sparse_hessian.jl")
include("forward.jl")
include("reverse.jl")
include("enzyme.jl")
include("di.jl")
include("predefined_backend.jl")
include("nlp.jl")

Expand Down
212 changes: 212 additions & 0 deletions src/di.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
struct DIADGradient{B, E} <: ADBackend
backend::B
prep::E
end

function DIADGradient(
nvar::Integer,
f,
ncon::Integer = 0,
c::Function = (args...) -> [];
x0::AbstractVector = rand(nvar),
backend = AutoReverseDiff(),
kwargs...,
)
prep = DifferentiationInterface.prepare_gradient(f, backend, x0)
return DIADGradient(backend, prep)
end

function gradient(b::DIADGradient, f, x)
g = DifferentiationInterface.gradient(f, b.prep, b.backend, x)
return g
end

function gradient!(b::DIADGradient, g, f, x)
DifferentiationInterface.gradient!(f, g, b.prep, b.backend, x)
return g
end

struct DIADJprod{B, E} <: ADBackend
backend::B
prep::E
end

function DIADJprod(
nvar::Integer,
f,
ncon::Integer = 0,
c::Function = (args...) -> [];
x0::AbstractVector = rand(nvar),
backend = AutoReverseDiff(),
kwargs...,
)
T = eltype(x0)
dy = similar(x0, ncon)
dx = ntuple(_ -> zero(T), nvar)
prep = DifferentiationInterface.prepare_pushforward(c, dy, backend, x0, dx)
return DIADJprod(backend, prep)
end

function Jprod!(b::DIADJprod, Jv, c, x, v, ::Val)
DifferentiationInterface.pushforward!(c, Jv, b.prep, b.backend, x, Tuple(v))
return Jv
end

struct DIADJtprod{B, E} <: ADBackend
backend::B
prep::E
end

function DIADJtprod(
nvar::Integer,
f,
ncon::Integer = 0,
c::Function = (args...) -> [];
x0::AbstractVector = rand(nvar),
backend = AutoReverseDiff(),
kwargs...,
)
T = eltype(x0)
dx = similar(x0, nvar)
dy = ntuple(_ -> zero(T), ncon)
prep = DifferentiationInterface.prepare_pullback(c, dx, backend, x0, dy)
return DIADJtprod(backend, prep)
end

function Jtprod!(b::DIADJtprod, Jtv, c, x, v, ::Val)
DifferentiationInterface.pullback!(c, Jtv, b.prep, b.backend, x, Tuple(v))
return Jtv
end

struct DIADJacobian{B, E} <: ADBackend
backend::B
prep::E
end

function DIADJacobian(
nvar::Integer,
f,
ncon::Integer = 0,
c::Function = (args...) -> [];
x0::AbstractVector = rand(nvar),
backend = AutoForwardDiff(),
kwargs...,
)
y = similar(x0, ncon)
prep = DifferentiationInterface.prepare_jacobian(c, y, backend, x0)
return DIADJacobian(backend, prep)
end

function jacobian(b::DIADJacobian, c, x)
J = DifferentiationInterface.jacobian(c, b.prep, b.backend, x)
return J
end

struct SparseDIADJacobian{B, E} <: ADBackend
backend::B
prep::E
end

function SparseDIADJacobian(
nvar::Integer,
f,
ncon::Integer = 0,
c::Function = (args...) -> [];
x0::AbstractVector = rand(nvar),
coloring_algorithm::AbstractColoringAlgorithm = GreedyColoringAlgorithm{:direct}(
postprocessing = true,
),
detector::AbstractSparsityDetector = TracerSparsityDetector(),
backend = AutoForwardDiff(),
kwargs...,
)
y = similar(x0, ncon)
sparse_backend = DifferentiationInterface.AutoSparse(backend, sparsity_detector=detector, coloring_algorithm=coloring_algorithm)
prep = DifferentiationInterface.prepare_jacobian(c, y, sparse_backend, x0)
return SparseDIADJacobian(sparse_backend, prep)
end

function jacobian(b::SparseDIADJacobian, c, x)
J = DifferentiationInterface.jacobian(c, b.prep, b.backend, x)
return J
end

struct DIADHvprod{B, E} <: ADBackend
backend::B
prep::E
end

function DIADHvprod(
nvar::Integer,
f,
ncon::Integer = 0,
c::Function = (args...) -> [];
x0::AbstractVector = rand(nvar),
backend = AutoReverseDiff(),
kwargs...,
)
T = eltype(x0)
tx = ntuple(_ -> zero(T), nvar)
prep = DifferentiationInterface.prepare_hvp(f, backend, x0, tx)
return DIADHvprod(backend, prep)
end

function Hvprod!(b::DIADHvprod, Hv, f, x, v, ::Val)
DifferentiationInterface.hvp!(f, Hv, b.prep, b.backend, x, Tuple(v))
return Hv
end

struct DIADHessian{B, E} <: ADBackend
backend::B
prep::E
end

function DIADHessian(
nvar::Integer,
f,
ncon::Integer = 0,
c::Function = (args...) -> [];
x0::AbstractVector = rand(nvar),
first_backend = AutoReverseDiff(),
second_backend = AutoForwardDiff(),
kwargs...,
)
backend = DifferentiationInterface.SecondOrder(second_backend, first_backend)
prep = DifferentiationInterface.prepare_hessian(f, backend, x0)
return DIADHessian(backend, prep)
end

function hessian(b::DIADHessian, f, x)
H = DifferentiationInterface.hessian(f, b.prep, b.backend, x)
return H
end

struct SparseDIADHessian{B, E} <: ADBackend
backend::B
prep::E
end

function SparseDIADHessian(
nvar::Integer,
f,
ncon::Integer = 0,
c::Function = (args...) -> [];
x0::AbstractVector = rand(nvar),
coloring_algorithm::AbstractColoringAlgorithm = GreedyColoringAlgorithm{:substitution}(
postprocessing = true,
),
detector::AbstractSparsityDetector = TracerSparsityDetector(),
first_backend = AutoReverseDiff(),
second_backend = AutoForwardDiff(),
kwargs...,
)
backend = DifferentiationInterface.SecondOrder(second_backend, first_backend)
sparse_backend = DifferentiationInterface.AutoSparse(backend, sparsity_detector=detector, coloring_algorithm=coloring_algorithm)
prep = DifferentiationInterface.prepare_hessian(f, backend, x0)
return SparseDIADHessian(sparse_backend, prep)
end

function hessian(b::SparseDIADHessian, f, x)
H = DifferentiationInterface.hessian(f, b.prep, b.backend, x)
return H
end
16 changes: 16 additions & 0 deletions src/predefined_backend.jl
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,27 @@ enzyme_backend = Dict(
:hessian_residual_backend => SparseEnzymeADHessian,
)

di_backend = Dict(
:gradient_backend => DIADGradient,
:jprod_backend => DIADJprod,
:jtprod_backend => DIADJtprod,
:hprod_backend => DIADHvprod,
:jacobian_backend => DIADJacobian, # SparseDIADJacobian,
:hessian_backend => DIADHessian, # SparseDIADHessian,
:ghjvprod_backend => EmptyADbackend,
:jprod_residual_backend => DIADJprod,
:jtprod_residual_backend => DIADJtprod,
:hprod_residual_backend => DIADHvprod,
:jacobian_residual_backend => DIADJacobian, # SparseDIADJacobian,
:hessian_residual_backend => DIADHessian, # SparseDIADHessian,
)

predefined_backend = Dict(
:default => default_backend,
:optimized => optimized_backend,
:generic => generic_backend,
:enzyme => enzyme_backend,
:di => di_backend,
)

"""
Expand Down
38 changes: 37 additions & 1 deletion test/runtests.jl
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,43 @@ for problem in NLPModelsTest.nls_problems
include("nls/problems/$(lowercase(problem)).jl")
end

include("utils.jl")
ReverseDiffAD(nvar, f) = ADNLPModels.ADModelBackend(
nvar,
f,
gradient_backend = ADNLPModels.ReverseDiffADGradient,
hprod_backend = ADNLPModels.ReverseDiffADHvprod,
jprod_backend = ADNLPModels.ReverseDiffADJprod,
jtprod_backend = ADNLPModels.ReverseDiffADJtprod,
jacobian_backend = ADNLPModels.ReverseDiffADJacobian,
hessian_backend = ADNLPModels.ReverseDiffADHessian,
)

function test_getter_setter(nlp)
@test get_adbackend(nlp) == nlp.adbackend
if typeof(nlp) <: ADNLPModel
set_adbackend!(nlp, ReverseDiffAD(nlp.meta.nvar, nlp.f))
elseif typeof(nlp) <: ADNLSModel
function F(x; nequ = nlp.nls_meta.nequ)
Fx = similar(x, nequ)
nlp.F!(Fx, x)
return Fx
end
set_adbackend!(nlp, ReverseDiffAD(nlp.meta.nvar, x -> sum(F(x) .^ 2)))
end
@test typeof(get_adbackend(nlp).gradient_backend) <: ADNLPModels.ReverseDiffADGradient
@test typeof(get_adbackend(nlp).hprod_backend) <: ADNLPModels.ReverseDiffADHvprod
@test typeof(get_adbackend(nlp).hessian_backend) <: ADNLPModels.ReverseDiffADHessian
set_adbackend!(
nlp,
gradient_backend = ADNLPModels.ForwardDiffADGradient,
jtprod_backend = ADNLPModels.GenericForwardDiffADJtprod(),
)
@test typeof(get_adbackend(nlp).gradient_backend) <: ADNLPModels.ForwardDiffADGradient
@test typeof(get_adbackend(nlp).hprod_backend) <: ADNLPModels.ReverseDiffADHvprod
@test typeof(get_adbackend(nlp).jtprod_backend) <: ADNLPModels.GenericForwardDiffADJtprod
@test typeof(get_adbackend(nlp).hessian_backend) <: ADNLPModels.ReverseDiffADHessian
end

include("nlp/basic.jl")
include("nlp/nlpmodelstest.jl")
include("nls/basic.jl")
Expand Down
Loading
Loading