[V3] Phase 1b: Refactor tests

[ocots]

Update tests implementation to automatise.

See the following documentation: https://control-toolbox.org/CTBase.jl/stable/test-coverage-guide.html.

See also this example: control-toolbox/CTModels.jl#239.

[ocots]

CTFlows Test Infrastructure Refactoring - Planning

Issue: #151 - [V3] Phase 1b: Refactor tests
Date: 2025-12-20
Status: Planning Complete ✅

TL;DR

Refactor CTFlows test infrastructure to use CTBase.run_tests pattern, enabling test filtering, automated coverage reporting, and following CTBase testing guide best practices. This will improve test iteration speed and maintainability while achieving ≥90% coverage target.

---

1. Overview

Goal

Transform CTFlows test suite from manual include() pattern to automated CTBase.run_tests() system with:

• Modular test execution with filtering capability
• Automated coverage reporting via test/coverage.jl
• Organized test structure following CTBase guide
• ≥90% test coverage for all modules

Key Features

• Test filtering: Run specific test groups via command-line arguments
• Coverage automation: Single command for execution + coverage report
• Modular structure: Each test file defines a standalone test function
• Documentation: Clear examples and workflows for contributors

References

• CTBase Testing Guide
• CTModels.jl Example
• CTBase.run_tests

---

2. Current State Analysis

Existing Test Structure

Current test/runtests.jl: Manual loop with include() and @eval

• ❌ No test filtering capability
• ❌ No coverage automation
• ❌ Hard to run individual test groups
• ✅ Tests are organized by module (good foundation)

Test Files (17 total):

• test_differential_geometry.jl ✅ (recently completed, 48 tests)
• test_types.jl
• test_concatenation.jl
• test_default.jl
• test_flow_function.jl
• test_flow_hamiltonian.jl
• test_flow_hamiltonian_vector_field.jl
• test_flow_vector_field.jl
• test_optimal_control_problem.jl
• test_saveat.jl
• test_aqua.jl

Missing:

• ❌ test/coverage.jl - Coverage automation script
• ❌ test/suite/ directory - Modular test organization
• ❌ Test functions wrapping @testset blocks

---

3. Technical Decisions

Decision	Choice	Rationale
Test runner	CTBase.run_tests()	Standard pattern, enables filtering
Test data structure	Tuple of Symbols	Per CTBase implementation (not Dict)
Builders required	filename_builder and funcname_builder	Required when using test_ prefix
Directory structure	test/suite/test_*.jl	Clean separation, recommended by CTBase
Coverage tool	CTBase.postprocess_coverage()	Automated, consistent with ecosystem
Minimum coverage	≥90%	CTBase recommendation
Test function pattern	function test_module_name()	Enables programmatic execution
Fake types location	Top-level in test files	Julia constraint (struct must be top-level)
Test file migration	Gradual, by module	Minimize disruption

---

4. Implementation Tasks

Phase 1: Infrastructure Setup

Task	Description
1.1	Create test/coverage.jl following CTBase pattern
1.2	Create test/suite/ directory structure
1.3	Update test/runtests.jl to use CTBase.run_tests()
1.4	Define available_tests dictionary in runtests.jl

Acceptance: Can run julia --project=. -e 'using Pkg; Pkg.test("CTFlows"; test_args=["differential_geometry"])'

---

Phase 2: Migrate Existing Tests

Task	Description
2.1	Wrap test_differential_geometry.jl content in function test_differential_geometry()
2.2	Move to test/suite/test_differential_geometry.jl
2.3	Migrate test_types.jl
2.4	Migrate test_concatenation.jl
2.5	Migrate test_default.jl
2.6	Migrate flow-related tests (4 files)
2.7	Migrate test_optimal_control_problem.jl
2.8	Migrate test_saveat.jl and test_aqua.jl

Acceptance: All tests pass with new structure, can run individually

---

Phase 3: Coverage and Documentation

Task	Description
3.1	Run coverage for each test group, identify gaps
3.2	Add tests to reach ≥90% coverage per module
3.3	Update README with test commands
3.4	Add coverage workflow documentation
3.5	Clean up old test files and backups

Acceptance: ≥90% coverage achieved, documented in README

---

5. Test File Structure Template

Recommended Pattern

# test/suite/test_module_name.jl

# ============================================================
# Fake types for unit testing (MUST be at top-level!)
# ============================================================

"""Fake model for testing module_name without dependencies."""
struct FakeModuleModel <: CTModels.AbstractModel
    # ...
end

# Overload necessary methods for fake model
# ...

# ============================================================
# Main test function
# ============================================================

function test_module_name()
    
    # ========================================================
    # Unit tests – helper logic (no side effects)
    # ========================================================
    @testset "module_name helpers" begin
        # Test pure functions, utilities
        @test helper_function(args) == expected
    end
    
    # ========================================================
    # Integration tests – actual functionality  
    # ========================================================
    @testset "module_name integration" begin
        # Test main API with fake/real models
        model = FakeModuleModel()
        result = main_function(model, args)
        @test result isa ExpectedType
    end
    
    # ========================================================
    # Edge cases and error handling
    # ========================================================
    @testset "module_name edge cases" begin
        @test_throws ErrorType function_with_validation(bad_args)
    end
end

Important

Julia Constraint: struct definitions MUST be at top-level, NOT inside functions.
Place all fake types before the function test_*() definition.

---

6. Updated test/runtests.jl

using Test
using CTFlows
using CTBase
using OrdinaryDiffEq
using Plots
using LinearAlgebra
using CTModels
import CTParser: CTParser, @def

const CTFlowsODE = Base.get_extension(CTFlows, :CTFlowsODE)

# For nested testset output formatting (used by individual test files)
const VERBOSE = true
const SHOWTIMING = true

# ============================================================
# Available test groups
# ============================================================
# Each test group corresponds to:
#   - A file: test/suite/test_<name>.jl
#   - A function: test_<name>()

available_tests = (
    :differential_geometry,
    :types,
    :concatenation,
    :default,
    :flow_function,
    :flow_hamiltonian,
    :flow_hamiltonian_vector_field,
    :flow_vector_field,
    :optimal_control_problem,
    :saveat,
    :aqua,
)

# ============================================================
# Run tests with CTBase
# ============================================================

CTBase.run_tests(;
    args=String.(ARGS),
    testset_name="CTFlows tests",
    available_tests=available_tests,
    filename_builder=name -> "test_$(name).jl",
    funcname_builder=name -> "test_$(name)",
    verbose=VERBOSE,
    showtiming=SHOWTIMING,
    test_dir=joinpath(@__DIR__, "suite"),
)

---

7. Coverage Setup

###Create test/coverage.jl

pushfirst!(LOAD_PATH, @__DIR__)

using Coverage
using CTBase

CTBase.postprocess_coverage(; root_dir=dirname(@__DIR__))

Coverage Commands

Run specific test with coverage:

julia --project=@. -e 'using Pkg; Pkg.test("CTFlows"; coverage=true, test_args=["differential_geometry"]); include("test/coverage.jl")'

Run all tests with coverage:

julia --project=@. -e 'using Pkg; Pkg.test("CTFlows"; coverage=true); include("test/coverage.jl")'

Output Example:

Coverage Summary:
─────────────────────────────────────────────────────────────
File                           Lines    Covered    Percentage
─────────────────────────────────────────────────────────────
src/differential_geometry.jl     210        195         92.9%
src/types.jl                     145        132         91.0%
...
─────────────────────────────────────────────────────────────
Total                            850        765         90.0%

Coverage Target

≥ 90% coverage for each module

Iteration Process

1. Run coverage for a test group
2. Identify uncovered lines
3. Add targeted tests
4. Re-run coverage
5. Repeat until ≥90%

---

8. Test Execution Examples

Run specific test group

julia --project=. -e 'using Pkg; Pkg.test("CTFlows"; test_args=["differential_geometry"])'

Run multiple groups

julia --project=. -e 'using Pkg; Pkg.test("CTFlows"; test_args=["differential_geometry", "types"])'

Run all tests

julia --project=. -e 'using Pkg; Pkg.test("CTFlows")'

Run with coverage

julia --project=@. -e 'using Pkg; Pkg.test("CTFlows"; coverage=true, test_args=["differential_geometry"]); include("test/coverage.jl")'

---

9. GitHub Workflow

PR Structure

Issue #151 ([V3] Phase 1b: Refactor tests) ← Epic
  ├── PR "Phase 1: Test Infrastructure Setup" → linked to #151
  ├── PR "Phase 2: Migrate Core Tests" → linked to #151
  └── PR "Phase 3: Coverage and Documentation" → closes #151

Implementation Checklist

• Phase 1: Infrastructure Setup

  • Create test/coverage.jl
  • Create test/suite/ directory
  • Update test/runtests.jl with CTBase.run_tests()
  • Define available_tests dictionary
  • Verify test filtering works

• Phase 2: Migrate Tests

  • Migrate differential_geometry (template)
  • Migrate types
  • Migrate concatenation
  • Migrate default
  • Migrate flow tests (4 files)
  • Migrate optimal_control_problem
  • Migrate saveat and aqua

• Phase 3: Coverage & Docs

  • Run coverage on all modules
  • Add tests to reach ≥90% per module
  • Update README with test commands
  • Document coverage workflow
  • Clean up old files

---

10. MVP (Minimum Viable Product)

MVP = Phase 1 + First migration (differential_geometry)

Includes:

• ✅ test/runtests.jl using CTBase.run_tests()
• ✅ test/coverage.jl for automated coverage
• ✅ test/suite/ directory structure
• ✅ test/suite/test_differential_geometry.jl migrated
• ✅ Working test filtering: julia ... test_args=["differential_geometry"]
• ✅ Working coverage: julia ... coverage=true; include("test/coverage.jl")

Validates:

• Infrastructure is correct
• Pattern can be replicated for other modules
• Coverage automation works
• Documentation is clear

---

11. Migration Guidelines

For Each Test File Migration

1. Create function wrapper

   function test_module_name()
       # Move all @testset blocks here
   end

2. Move fake types to top-level (if any)

   # BEFORE function definition
   struct FakeModel <: AbstractModel end

3. Preserve test organization

   • Keep @testset structure
   • Maintain test descriptions
   • Keep helper functions if module-specific

4. Update available_tests in runtests.jl

5. Test independently

   julia --project=. -e 'using Pkg; Pkg.test("CTFlows"; test_args=["module_name"])'

6. Run coverage

   julia --project=@. -e 'using Pkg; Pkg.test("CTFlows"; coverage=true, test_args=["module_name"]); include("test/coverage.jl")'

---

12. Success Criteria

Infrastructure

• CTBase.run_tests() operational
• Test filtering works for all modules
• Coverage automation functional
• Clear documentation in README

Tests

• All existing tests migrated
• All tests pass in new structure
• Tests runnable individually
• Tests runnable as complete suite

Coverage

• ≥90% coverage per module
• Coverage reports automated
• Uncovered lines documented/justified

Documentation

• README updated with test commands
• Coverage workflow documented
• Migration pattern clear for future contributors

---

13. Estimated Effort

Phase	Tasks	Effort	Risk
Phase 1	Infrastructure (4 tasks)	2-3 hours	Low
Phase 2	Migration (8 test files)	4-6 hours	Medium
Phase 3	Coverage + Docs (5 tasks)	3-4 hours	Medium
Total	17 tasks	9-13 hours	Medium

Risk Factors:

• Breaking changes during migration
• Coverage gaps requiring new test design
• Edge cases in existing tests
• Extension tests (CTFlowsODE) may need updates

---

14. Post-Implementation

Benefits

• ✅ Fast test iteration (run only what changed)
• ✅ Automated coverage tracking
• ✅ Better contributor experience
• ✅ Consistent with CTBase ecosystem
• ✅ Easier to maintain and extend

Maintenance

• Update available_tests when adding new modules
• Run coverage regularly during development
• Keep test structure documentation current
• Review coverage reports in PRs

---

15. References

Documentation

• CTBase Testing Guide
• CTBase.run_tests API
• Coverage.jl

Examples

• CTModels.jl #239
• CTBase test structure

Tools

• CTBase >= v0.17.2 (for postprocess_coverage)
• Coverage.jl (for coverage collection)
• Test.jl (standard library)

---

Planning Status: ✅ Complete and ready for implementation

Next Step: Review this plan, then begin Phase 1 implementation