No-op code & docs cleanups #245

alyst · 2025-02-04T07:22:24Z

These are rather trivial commits cherry-picked from #193 that don't change anything essential, but once they are in, it should be easier to manage the rest.

The changes are:

code formatter whitespace fixes for the .md files
move some using directives from the individual unit test .jl files to the parent .jl file
in RAMSymbolic rename xxx_function to xxx_eval! as that better follows Julia naming conventions and should make the code a bit more readable (and shorter)
cleanups to the fitmeasures that simplify some method signatures (remove unused type parameters).
This is a bit less trivial set of changes than the rest, but in the end it does not really change how the functions are calculated.
Except for switching p-values calculation to use ccdf(x) instead of 1 - cdf(x) -- this could potentially give more precise results for p-values close to 0.

codecov · 2025-02-04T07:30:03Z

Codecov Report

❌ Patch coverage is 82.75862% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.24%. Comparing base (8b0f880) to head (acd1748).
⚠️ Report is 57 commits behind head on devel.

Files with missing lines	Patch %	Lines
src/optimizer/abstract.jl	10.00%	9 Missing ⚠️
src/additional_functions/helper.jl	0.00%	3 Missing ⚠️
ext/SEMNLOptExt/NLopt.jl	85.71%	2 Missing ⚠️
src/additional_functions/params_array.jl	33.33%	2 Missing ⚠️
src/frontend/fit/fitmeasures/chi2.jl	92.85%	2 Missing ⚠️
src/implied/RAM/symbolic.jl	93.10%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##            devel     #245      +/-   ##
==========================================
- Coverage   72.94%   72.24%   -0.71%     
==========================================
  Files          50       51       +1     
  Lines        2218     2212       -6     
==========================================
- Hits         1618     1598      -20     
- Misses        600      614      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

alyst · 2025-03-22T07:17:34Z

I've rebased the PR after the #253 got merged.

@Maximilian-Stefan-Ernst I have also noticed that you have implemented #246.
I am not sure I understand what was exactly the reason for these changes -- I guess it was related to running some examples/tests, but if that's the case we can fix it differently.
To me this change defeats the purpose of using extension mechanism for supporting different optimization engines as it moves the SemOptimizer subtypes back into the main package.
There is no strong reason to make engine-specific SemOptimizer subtypes public and recommend the user constructing them explicitly, especially since these constructors take all their input as keyword arguments.
Yes, before #246 the SemOptimizerEngine type was only available within the extension and was not exposed upon extension activation,
but SemOptimizer(engine = ...) => SemOptimizer{engine}(...) => SemOptimizerEngine(...) mechanism can handle SemOptimizerEngine construction just fine.
It is a standard pattern for implementing Julia extensions.
Also, I think for the user it is easier to memorize the SemOptimizer(engine = ...) construct, and it also makes it easier to switch the engines.

So I have added commits that revert #246 in this PR, but also updated the optimization documentation to be in sync with the actual opt. engine API:

SemOptimizationOptim(), SemOptimizationNLopt() etc in the docs are replaced with SemOptimizer(engine = <engine_name>, args...).
I have updated the fallback SemOptimizer(engine = ...) constructor to inform the user, which packages should be loaded for NLopt and ProximalAlgorithms, if that was one of the reasons for Fix extensions #246.
For NLopt I have replaced NLoptConstraint(f = func, tol = tol) with func => tol.
That simplifies the syntax and avoids defining NLoptContraint type, which is only really used for SemOptimizerNLopt construction.
I have also allowed specifying just the function without the tolerance -- in that case the tolerance will be taken from the constraint_tol parameter.
In the documentation I have changed the term "backend" into "engine" to match the SemOptimizer keyword, but if you have strong preference for "backend",
I can revert it and rename the "engine" keyword into "backend".
I have also simplified the examples: since the optimizer is now separate from the model, we don't need to duplicate the model if we want to fit it with another optimizer.

Please let me know if you have any concerns regarding these changes, I will be happy to address them.

alyst · 2025-11-24T19:30:29Z

@Maximilian-Stefan-Ernst @brandmaier @aaronpeikert In the recent months the merging of some of the changes from my staging branch (PR #193) into the main branch was dormant. But I would be still interested in it, if you think it will be beneficial for the SEM.jl. I mention it, because I see there are new issues/PRs, which could at least partially be addressed in PR #193. Thank you!

alyst · 2026-01-21T08:34:27Z

The FormatCheck action updates were extracted to PR #296.
It probably has to be merged first.

Maximilian-Stefan-Ernst · 2026-01-22T12:55:14Z

Thanks a lot! I merged #296 - is this PR ready for review, or is there anything else you would like to add before?

also remove unused _symbolic fields

This reverts commit 9729819.

This reverts commit 81a4bd9.

This reverts commit f0df653.

It is a simple and intuitive syntax and avoids declaring new types. Also allow specifying default constraint tolerance as `constraint_tol`.

use SemOptimizer(engine = :NLopt) instead of SemOptimizerNLopt() as this is a more universal scheme

alyst · 2026-01-22T18:54:12Z

@Maximilian-Stefan-Ernst Thank you! I've just rebased this branch to the latest devel, but somehow GitHub does not pick up the FormatCheck.yml changes (still uses v1).
Anyway, this PR should be ready for the review.

Maximilian-Stefan-Ernst · 2026-01-26T11:21:06Z

Alright, I'll review - I will also try to remind myself why I made the changes related to package extensions and write something about it.

W.r.t. the FormatCheck action - I believe changes to workflows only take effect once they are merged into main - I added the changes to main now, so it should pick up.

Maximilian-Stefan-Ernst · 2026-01-26T12:36:08Z

Okay, w.r.t. the package extensions:

I believe the problem was that there was no way to access types defined in extensions at the time. This seems to be fixed now, but it is still cumbersome (https://discourse.julialang.org/t/having-trouble-using-types-defined-in-extension-modules/97260):

using StructuralEquationModels, NLopt

?SemOptimizerNLopt

Couldn't find SemOptimizerNLopt
Perhaps you meant SemOptimizerEmpty
  No documentation found.

  Binding SemOptimizerNLopt does not exist.

m = Base.get_extension(StructuralEquationModels, :SEMNLOptExt)
?m.SemOptimizerNLopt

So out of the box, this leads to some problems - without explicitely using get_extension, users can not access documentation on the extensions, and they can not define methods on types from extensions.

So I thought the main objective of having extensions is to avoid unnecessary dependencies - therefore I moved the types that are defined to the main package, so they are accessible, and their definition does not depend on the dependencies. Definitions of new methods, however, remained in the extension because they actually depend on additonal packages.

However, I really like your changes to the documentation, and the improved syntax for defining constraints, and I would definitely keep those. I am also happy with reverting the changes and moving the types back to the extension, the only thing we would need to fix is the accessibility of the documentation.

I also pushed these changes to another branch because I did not manage to manually trigger the documentation build workflow on this PR, but it is now here (https://github.com/StructuralEquationModels/StructuralEquationModels.jl/actions/runs/21357427498/job/61468060053) and it also seems to have the issue of accessing the documentation of types defined in extensions.

So the only thing I can think of atm would be to move the documentation of all extensions to SemOptimizer - but these docs are pretty long, and that might be quite overwhelming.

alyst · 2026-01-26T18:31:02Z

Okay, w.r.t. the package extensions:
I believe the problem was that there was no way to access types defined in extensions at the time. This seems to be fixed now, but it is still cumbersome (https://discourse.julialang.org/t/having-trouble-using-types-defined-in-extension-modules/97260):
using StructuralEquationModels, NLopt

?SemOptimizerNLopt

Couldn't find SemOptimizerNLopt

Yes, initially, I had the same confusion about Julia extensions.
It looks like they don't allow defining types -- either because of the implementation constraints or by design to limit their scope.
But in the case of the optimizer engines we actually don't need to expose the types like SemOptimizerNLOpt -- this is an implementation detail.
The user have to use the SemOptimizer(engine = ...) call, which also makes the API more flexible.

So out of the box, this leads to some problems - without explicitely using get_extension, users can not access documentation on the extensions, and they can not define methods on types from extensions.

I totally agree, the extensions need to provide the docstrings for the specific engine parameters, but currently in this PR it does not happen as it documents
these internal types/ctors. This definitely needs to be fixed. I have tried to fix it, but I am running into some DocStringExtensions.jl issues, so I have to research a bit more.

Which kind of use case you have in mind for allowing the users to define methods on the types of extensions?
Within the extensions it is possible to define the methods with engine-specific types as parameters.
It could be used for engine-specific convergence statistics etc.

However, I really like your changes to the documentation, and the improved syntax for defining constraints, and I would definitely keep those. I am also happy with reverting the changes and moving the types back to the extension, the only thing we would need to fix is the accessibility of the documentation.

+1 for fixing the docs.

So the only thing I can think of atm would be to move the documentation of all extensions to SemOptimizer - but these docs are pretty long, and that might be quite overwhelming.

Yes, I agree -- it might be too long and unspecific. I'll take a look how this is handled in other packages.
For example, the SemOptimizer{:engine}() docstrings could be very short, providing just a link to the proper docs.
It also solves the problem that the user has to discover the backend-specific types (SemOptimizerNLOpt) first to get their documentation.

To improve the discovery, I can also try adding the function optimizer_engines(), which would return the vector of symbols for the engines that are currently available.

github-actions · 2026-01-26T18:32:29Z

ext/SEMNLOptExt/NLopt.jl

+        inequality_constraints = nothing,
+        constraint_tol::Number = 0.0,
+        kwargs...)
+


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

!isa(inequality_constraints, NamedTuple) ||

github-actions · 2026-01-26T18:32:29Z

src/additional_functions/helper.jl


 # computes A*S*B -> C, where ind gives the entries of S that are 1
-function sparse_outer_mul!(C, A, B, ind) 
+function sparse_outer_mul!(C, A, B, ind)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

function sparse_outer_mul!(C, A, B, ind)

function sparse_outer_mul!(C, A, B, ind)

github-actions · 2026-01-26T18:32:29Z

src/additional_functions/helper.jl


 # computes A*∇m, where ∇m ind gives the entries of ∇m that are 1
-function sparse_outer_mul!(C, A, ind) 
+function sparse_outer_mul!(C, A, ind)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

function sparse_outer_mul!(C, A, ind)

function sparse_outer_mul!(C, A, ind)

github-actions · 2026-01-26T18:32:29Z

src/additional_functions/helper.jl


 # computes A*S*B -> C, where ind gives the entries of S that are 1
-function sparse_outer_mul!(C, A, B::Vector, ind) 
+function sparse_outer_mul!(C, A, B::Vector, ind)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

function sparse_outer_mul!(C, A, B::Vector, ind)

function sparse_outer_mul!(C, A, B::Vector, ind)

github-actions · 2026-01-26T18:32:29Z

src/frontend/StatsAPI.jl

+    ArgumentError(
+        "StructuralEquationModels does not support the `CoefTable` interface; see [`ParameterTable`](@ref) instead.",
+    ),
+)


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

)

coeftable(model::AbstractSem; level::Real = 0.95) = throw(

ArgumentError(

"StructuralEquationModels does not support the `CoefTable` interface; see [`ParameterTable`](@ref) instead.",

),

)

github-actions · 2026-01-26T18:32:30Z

test/examples/proximal/l0.jl

-        StructuralEquationModels.minimum(fit_prox) - StructuralEquationModels.minimum(sem_fit),
+        StructuralEquationModels.minimum(fit_prox) -
+        StructuralEquationModels.minimum(sem_fit),
    ) < 1.0


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

) < 1.0

StructuralEquationModels.minimum(fit_prox) -

StructuralEquationModels.minimum(sem_fit),

github-actions · 2026-01-26T18:32:30Z

test/unit_tests/StatsAPI.jl

+model = Sem(specification = partable, data = data)
 model_fit = fit(model)

 @testset "params" begin


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

model = Sem(specification = partable, data = data)

model_fit = fit(model)

@testset "params" begin

model = Sem(specification = partable, data = data)

github-actions · 2026-01-26T18:32:30Z

test/unit_tests/model.jl

 end

-
 ram_matrices =


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

ram_matrices =

github-actions · 2026-01-26T18:32:31Z

test/unit_tests/unit_tests.jl

    "specification" => "SemSpecification",
    "model" => "Sem model",
-    "StatsAPI" => "StatsAPI"
+    "StatsAPI" => "StatsAPI",


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

"StatsAPI" => "StatsAPI",

"StatsAPI" => "StatsAPI",

github-actions · 2026-01-26T18:32:31Z

test/unit_tests/unit_tests_interactive.jl

    @warn "Error initializing Test Env" exception=(e, catch_backtrace())
 end
-include("unit_tests.jl")
+include("unit_tests.jl")


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

include("unit_tests.jl")

include("unit_tests.jl")

github-actions · 2026-01-26T18:32:35Z

src/additional_functions/params_array.jl

[JuliaFormatter] _{reported by reviewdog 🐶}

StructuralEquationModels.jl/src/additional_functions/params_array.jl

Lines 202 to 206 in acd1748

function sparse_materialize(::Type{T}, arr::ParamsMatrix, params::AbstractVector) where {T}

nparams(arr) == length(params) || throw(

DimensionMismatch(

"Number of values ($(length(params))) does not match the number of parameter ($(nparams(arr)))",

),

github-actions · 2026-01-26T18:32:36Z

src/additional_functions/params_array.jl

[JuliaFormatter] _{reported by reviewdog 🐶}

StructuralEquationModels.jl/src/additional_functions/params_array.jl

Lines 254 to 255 in acd1748

for i in first_i:last_i

if isempty(param_occurences_range(arr, i))

github-actions · 2026-01-26T18:32:37Z

src/frontend/StatsAPI.jl

[JuliaFormatter] _{reported by reviewdog 🐶}

StructuralEquationModels.jl/src/frontend/StatsAPI.jl

Lines 16 to 20 in acd1748

function params!(out::AbstractVector, partable::ParameterTable, col::Symbol = :estimate)

(length(out) == nparams(partable)) || throw(

DimensionMismatch(

"The length of parameter values vector ($(length(out))) does not match the number of parameters ($(nparams(partable)))",

),

github-actions · 2026-01-26T18:32:38Z

test/examples/helper.jl

[JuliaFormatter] _{reported by reviewdog 🐶}

StructuralEquationModels.jl/test/examples/helper.jl

Lines 93 to 98 in acd1748

expected =

StructuralEquationModels.lavaan_params(partable_lav, partable, lav_col, lav_group)

@test !any(isnan, actual)

@test !any(isnan, expected)

if skip # workaround skip=false not supported in earlier versions

alyst force-pushed the noop_cleanups branch from c65c8a9 to 0c7d594 Compare March 22, 2025 06:39

alyst closed this Apr 26, 2025

alyst reopened this Apr 26, 2025

alyst force-pushed the noop_cleanups branch from 0c7d594 to 79d2482 Compare January 20, 2026 23:20

alyst closed this Jan 21, 2026

alyst reopened this Jan 21, 2026

alyst and others added 19 commits January 22, 2026 10:34

minus2ll(): cleanup method signatures

e1b4bc3

fix chi2

1071533

fix RMSEA

1dd08dd

p_values(): use ccdf()

6cb3753

RAM: don't need to copy (I-A)

bb3df99

EM: move code refs to docstring

a5a553e

fix batch_sym_inv_updates() ws

affd75f

RAMSymbolic: rename _func to _eval!

1e60b16

also remove unused _symbolic fields

md ws fixes

3ef44c5

test/Proximal: move usings to the central file

ef5a22e

tests: move usings in the top file

d712b40

Revert "fix Proximal extension"

170e3a7

This reverts commit 9729819.

Revert "fix NLopt extension"

0a98e83

This reverts commit 81a4bd9.

Revert "fix exporting structs from package extensions"

41b6477

This reverts commit f0df653.

types.jl: move SemOptimizer API into abstract.jl

7c422c7

NLoptResult should not be mutable

ab47f49

SemNLOpt: use f or f => tol pair for constraints

a86bbef

It is a simple and intuitive syntax and avoids declaring new types. Also allow specifying default constraint tolerance as `constraint_tol`.

NLopt: update/simplify docs

5125e71

use SemOptimizer(engine = :NLopt) instead of SemOptimizerNLopt() as this is a more universal scheme

Optim.md: SemOptimizerOptim => SemOptiimizer

c425486

Alexey Stukalov added 13 commits January 22, 2026 10:34

documentation.jl: fix formatting

8883e60

EnsParTable: fix formatting

21da001

ParTable: fix formatting

89a4270

generic: fix formatting

681dd97

missing: fix formatting

fc822d9

abstract: fix formatting

326a130

optim: fix ws

4ad3de1

test/model: fix ws

08dcb2f

test/proximal/l0: fix ws

e5d0d65

test/multigroup: fix ws

bdeafe9

test/StatsAPI: fix ws

b549ef1

tests/unit_test_iactive: fix ws

22d1d13

tests/unit_tests: fix formatting

5ef1f71

alyst force-pushed the noop_cleanups branch from 7299ba5 to 5ef1f71 Compare January 22, 2026 18:36

alyst closed this Jan 22, 2026

alyst reopened this Jan 22, 2026

fix formattng

acd1748

alyst closed this Jan 22, 2026

alyst reopened this Jan 22, 2026

alyst closed this Jan 26, 2026

alyst reopened this Jan 26, 2026

github-actions bot reviewed Jan 26, 2026

View reviewed changes

	function sparse_outer_mul!(C, A, B, ind)
	function sparse_outer_mul!(C, A, B, ind)

	function sparse_outer_mul!(C, A, ind)
	function sparse_outer_mul!(C, A, ind)

	function sparse_outer_mul!(C, A, B::Vector, ind)
	function sparse_outer_mul!(C, A, B::Vector, ind)

	) < 1.0
	StructuralEquationModels.minimum(fit_prox) -
	StructuralEquationModels.minimum(sem_fit),

	function sparse_materialize(::Type{T}, arr::ParamsMatrix, params::AbstractVector) where {T}
	nparams(arr) == length(params) \|\| throw(
	DimensionMismatch(
	"Number of values ($(length(params))) does not match the number of parameter ($(nparams(arr)))",
	),

	for i in first_i:last_i
	if isempty(param_occurences_range(arr, i))

	function params!(out::AbstractVector, partable::ParameterTable, col::Symbol = :estimate)
	(length(out) == nparams(partable)) \|\| throw(
	DimensionMismatch(
	"The length of parameter values vector ($(length(out))) does not match the number of parameters ($(nparams(partable)))",
	),

	expected =
	StructuralEquationModels.lavaan_params(partable_lav, partable, lav_col, lav_group)
	@test !any(isnan, actual)
	@test !any(isnan, expected)

	if skip # workaround skip=false not supported in earlier versions

No-op code & docs cleanups #245

Are you sure you want to change the base?

No-op code & docs cleanups #245

Uh oh!

Conversation

alyst commented Feb 4, 2025

Uh oh!

codecov bot commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alyst commented Mar 22, 2025

Uh oh!

alyst commented Nov 24, 2025

Uh oh!

alyst commented Jan 21, 2026

Uh oh!

Maximilian-Stefan-Ernst commented Jan 22, 2026

Uh oh!

alyst commented Jan 22, 2026

Uh oh!

Maximilian-Stefan-Ernst commented Jan 26, 2026

Uh oh!

Maximilian-Stefan-Ernst commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alyst commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Feb 4, 2025 •

edited

Loading

Maximilian-Stefan-Ernst commented Jan 26, 2026 •

edited

Loading

alyst commented Jan 26, 2026 •

edited

Loading