`init/solve` structure, solver caches, tests, naming

[SteffenPL]

Sorry for all the spam (let me know if I should write fewer issues) 😝

Here are some ideas for next steps. (Obviously, you have more experience, I'm happy to get advice). We could

• introduce the classical init/solve structure
• add more tests, especially for AD
• caches for algorithms

For me, especially tests would be helpful as I don't have a good idea of what kind of inputs/interfaces we can assume.
For example, in the linear case, do we only assume a linear AbstractSciMLOperator? Do solvers need to be compatible with Duals? (or will the differentiation always be handled externally, like currently in adjoint.jl which seems mostly algorithm independent).

A few minor things:

• Naming conventions: For linear problems, it seems that the initial guess is called u0, which is confusing since in the solution type it is called z. Would it be an option to rename it to z0? On the other hand, for the mixed and nonlinear case u is maybe a better variable name. So, one could either always call the unknown u (linear/mixed/nonlinear) or have z (for linear) and u (for mixed/nonlinear).
• Tolerance and error estimates: I added error_estimate for the PSOR type solvers. In general, it would be nice to have a common way to set the tolerance of an algorithm. Should it be reltol/abstol or just one tol parameter?
• The field residual feels to be a bit ambiguous. Different algorithms might have different choices for the residuals (depending on the nonlinear functions). Should it return NaN if an algorithm has no residual for an algorithm (like for PSOR) or should it be an error estimator in that case?

[avik-pal]

Opening issues is the right way forward, IMO. It is just a better way to track progress.

For me, especially tests would be helpful as I don't have a good idea of what kind of inputs/interfaces we can assume.

I agree, I was testing the package locally, but now I will convert them to proper unit tests and add them to the repo.

in the linear case, do we only assume a linear AbstractSciMLOperator?

Yeah this needs to be documented. Not all the solvers currently can take a linear sciml operator, to ensure compatibility with the non linear solve adjoint.

one could either always call the unknown u (linear/mixed/nonlinear)

I prefer this option (unless ofcouse there is strong objection against it)

Should it be reltol/abstol or just one tol parameter?

We should probably do reltol/abstol to ensure compatibility the rest of SciMLVerse

residual

I think it is fine to return an error estimator if there is no residual perse. The other alternative could be having a stats / diagnostics field, and different solvers can pass a NamedTuple in it with its own internal diagnostics.

[ChrisRackauckas]

to ensure compatibility with the non linear solve adjoint.

We are fixing SciMLOperators to have nice adjoints. That's part of the plan, it's just taking a bit.

[ChrisRackauckas]

Opening issues is the right way forward, IMO. It is just a better way to track progress.

👍