Bring the legacy test suite to green (~540 failures outside the blocking layer)

[jeremymanning]

Overview

Following the major refactor in #307, we need to systematically audit our entire test suite to achieve a 100% pass rate while using tests to guide development.

Current Status

• 108 test files have been updated with correct import paths (orchestrator. → src.orchestrator.)
• Many tests may be obsolete due to architectural changes in refactor Major refector #307
• Some tests are broken but should work with current architecture
• Failing tests should guide toolbox improvements and missing functionality

Systematic Approach

Phase 1: Test Categorization

• Run complete test suite and capture all failures
• Categorize each failing test as:
  • OBSOLETE: No longer relevant after refactor, should be removed
  • BROKEN: Should work with current architecture, needs repair
  • GUIDE: Reveals missing functionality that needs implementation

Phase 2: Test Cleanup

• Remove all OBSOLETE tests that no longer apply to current architecture
• Document removed tests and reasoning

Phase 3: Test Repair

• Fix all BROKEN tests to work with current architecture
• Update test assertions to match new API contracts
• Ensure tests use correct import paths and class structures

Phase 4: Test-Driven Development

• Use GUIDE tests to implement missing functionality
• Create new GitHub issues for large-scale implementations
• Fix small missing pieces immediately

Phase 5: Validation

• Achieve 100% test pass rate locally
• Verify CI passes across all platforms (Ubuntu, macOS, Windows)
• Verify CI passes across all Python versions (3.11, 3.12)

Success Criteria

• 100% test pass rate locally and in CI
• All obsolete tests removed and documented
• All broken tests repaired or converted to implementation guides
• Test suite guides future development effectively

Related Issues

• Closes: orchestrator-api-completion epic
• Related: Major refector #307 (major refactor)
• Builds on: Import path fixes across 108 test files

[jeremymanning]

No longer relevant after Claude Skills refactor (issue #426)

[jeremymanning]

Re-verified: still far from met, and this is now the umbrella issue for test-suite health.

Current measured state on main (isolated py3.12 venv with .[dev] installed):

• 2,981 tests collect, 0 collection errors
• Blocking layer (unit/contract/e2e): 179 passed, 12 skipped — green
• The remaining ~2,800 are NOT green — roughly 540 failed / 110 errored when last measured

Collection passing is not the suite passing. Only the marked blocking layer gates CI; the rest runs in a non-blocking job specifically so this backlog stays visible rather than being hidden.

Also note commits 704e10e and 732b7fe ("Add module-level skips to tests with removed providers") made parts of the suite green by skipping, not by fixing — those skips are in scope here.

Absorbs #356 and #351, which described the same backlog from different angles.

---

Reopened during a repository-wide issue audit; closed 2025-10-18 as superseded by the Claude Skills refactor, which did not ship (see #426).