[EPIC] DI + Monorepo: Unified architecture transformation

[bashandbone]

Overview

Transform CodeWeaver's architecture through integrated Dependency Injection and monorepo structure, reducing circular dependencies by 75% and enabling clean package separation.

The Discovery

Critical Insight: The DI architecture eliminates most circular dependencies that block monorepo split. These should be implemented together, not sequentially.

Key Finding: DI factories replace manual registry access, which is the root cause of provider → engine → config circular dependencies.

The Problem

Current architecture has:

• 164 circular dependencies between modules
• Manual provider instantiation scattered throughout codebase
• 40+ providers with verbose setup
• Hard to test (manual mocking required)
• Will get worse as provider count grows to 100+

The Solution: Integrated DI + Monorepo

Phase 1: DI Foundation + Prep (Week 1)

• Build core DI infrastructure (container, Depends(), factories)
• Extract ready packages (tokenizers, daemon)
• Move SearchResult types to core
• Outcome: DI system ready, 2 packages extracted

Phase 2: DI Integration (Week 2)

• Migrate core services to DI
• Eliminates 120-130 violations (75%)
• Breaks circular dependencies
• Validates package boundaries
• Outcome: Clean dependency flow established

Phase 3: Monorepo Structure (Week 3)

• Organize into 9 packages with uv workspace
• All packages build independently
• Remaining violations: ~30-40 (down from 164)
• Outcome: Clean monorepo structure

How DI Solves Monorepo Blockers

Problem: Registry Circular Dependencies

# Before: Manual registry creates coupling
from codeweaver.common.registry import get_provider_registry
registry = get_provider_registry()
provider = registry.get_embedding_provider()

# After: DI factory handles complexity
from codeweaver.di.providers import EmbeddingDep

class MyService:
    def __init__(self, embedding: EmbeddingDep):
        self.embedding = embedding  # Clean!

Result: ✅ Eliminates providers → engine dependency entirely

Violations Eliminated by DI

Original: 164 violations
After DI: ~30-40 violations (75% reduction)

Eliminated:

• providers → engine (20 imports)
• telemetry → engine (3 imports)
• telemetry → config (3 imports)
• engine → CLI (5 imports)
• config → CLI (multiple)

Remaining (manual fixes):

• core → utils (9 imports) - Move utilities
• semantic → utils (4 imports)
• providers → agent_api (4 imports) - Move types

Implementation Phases

Phase 1: Foundation + Prep (#117)

• Core DI infrastructure
• Extract tokenizers, daemon
• Move SearchResult to core
• Risk: Low | Duration: 5-7 days

Phase 2: Core Migration (#118)

• Migrate Indexer, search services
• Break circular dependencies
• Update tests with DI overrides
• Risk: Medium | Duration: 7-10 days

Phase 3: Monorepo Structure (NEW ISSUE NEEDED)

• Organize into packages/
• Build system setup
• Final validation
• Risk: Low | Duration: 5-7 days

Phase 4: pydantic-ai Integration (#119)

• Agent DI support
• Data provider injection
• Risk: Medium

Phase 5: Advanced Features (#120)

• Health checks, telemetry
• Plugin system
• Risk: Low

Phase 6: Cleanup (#121)

• Deprecate old patterns
• Simplify/eliminate registry
• Risk: Low

Timeline

Total Duration: 2-3 weeks for core transformation
Extended: Additional 2-3 weeks for advanced features

Week 1: DI Foundation + Prep
Week 2: DI Integration (breaks dependencies)
Week 3: Monorepo Structure (natural organization)

Benefits

Immediate (After Phase 2)

• 60-70% less boilerplate in dependency management
• 75% reduction in circular dependencies
• Better testing with clean override mechanism
• Backward compatible during migration

Long-term

• Scalability: Adding providers becomes mechanical
• Maintainability: Clear package boundaries
• Type safety: Full inference and checking
• FastAPI alignment: Familiar patterns

Monorepo Structure (After Phase 3)

packages/
  codeweaver-core/          # Types, exceptions, DI
  codeweaver-tokenizers/    # ✅ Extracted
  codeweaver-daemon/        # ✅ Extracted
  codeweaver-utils/         # Common utilities
  codeweaver-semantic/      # Semantic chunking
  codeweaver-telemetry/     # Telemetry (DI-enabled)
  codeweaver-providers/     # All provider implementations
  codeweaver-engine/        # Engine, config (DI-enabled)
  codeweaver/               # CLI, server, MCP, agent_api

Success Criteria

Technical

• DI container fully functional
• All core services use DI
• Dependency violations < 50 (down from 164)
• All packages build independently
• Zero circular dependencies between packages
• Tests pass with 100% success rate

Quality

• 60-70% reduction in boilerplate
• Test code 50% less verbose
• New provider integration < 1 hour
• Performance within 5% of baseline

Related Issues

This work will obsolete or significantly simplify:

• Register default services in services registry #84 (Register default services - DI handles)
• Improve EmbeddingRegistry: configurable size, dynamic sizing, persistence #100 (EmbeddingRegistry improvements - registry becomes thin layer)
• Integrate Indexer with services registry and implement persistence loading #103 (Indexer service integration - DI replaces approach)

Documentation

See comprehensive analysis in monorepo planning docs:

• INTEGRATED_DI_MONOREPO_STRATEGY.md - Main strategy
• DI_IMPACT_VISUALIZATION.md - Visual impact
• plans/dependency-injection-architecture-plan.md - Original DI plan

Constitutional Compliance

✅ Proven Patterns (Principle II): Channels FastAPI's DI system
✅ Evidence-Based (Principle III): Incremental, testable phases
✅ Simplicity (Principle V): Hides complexity in factories
✅ AI-First Context (Principle I): Self-documenting dependencies

Key Insight

Your DI architecture plan is the missing piece that makes monorepo split elegant.

Combined Strategy:

• DI eliminates 75% of violations
• Monorepo split becomes trivial
• Better architecture (DI is valuable independently)
• 2-3 weeks instead of 3-4 weeks

Recommendation: PROCEED with integrated DI + Monorepo strategy

[github-actions]

Claude finished Adam Poulemanos (@bashandbone)'s task —— View job

---

Epic Issue #116 Analysis Complete

---

Executive Summary

This is a well-structured epic proposing a significant architectural improvement to replace manual provider instantiation with FastAPI-inspired dependency injection. The proposal is constitutionally sound and addresses real technical debt, but has critical gaps in planning documentation that need attention before implementation begins.

---

Issue Structure Assessment

✅ Strengths

Epic Organization:

• Clear 5-phase breakdown with risk assessment
• Single sub-issue created ([DI Phase 1] Foundation + monorepo prep (tokenizers, daemon, SearchResult) #117 for Phase 1)
• Well-defined benefits (60-70% boilerplate reduction)
• Migration strategy with version timeline (v0.2 → v0.4)
• Code comparison examples showing before/after

Constitutional Compliance:

• ✅ Principle II (Proven Patterns): Channels FastAPI's DI system
• ✅ Principle III (Evidence-Based): Incremental, testable phases
• ✅ Principle V (Simplicity): Hides complexity in factories
• ✅ Principle I (AI-First Context): Self-documenting dependencies

❌ Critical Gaps

Missing Planning Documents:
The epic references 5 planning documents that do not exist in the plans/ directory:

• DI_ARCHITECTURE_SUMMARY.md
• dependency-injection-architecture-plan.md (claimed to be 32KB)
• DI_ARCHITECTURE_DIAGRAMS.md
• DI_IMPLEMENTATION_CHECKLIST.md
• DI_PROVIDER_EXAMPLES.md

Missing Sub-Issues:
Only Phase 1 (#117) exists. Missing:

• [DI Phase 2] Core migration - Breaking 75% of circular dependencies #118 - Phase 2: Core Migration
• [DI Phase 4] pydantic-ai integration (agents + data sources) #119 - Phase 3: pydantic-ai Integration
• [DI Phase 5] Advanced features (health, telemetry, plugins) #120 - Phase 4: Advanced Features
• [DI Phase 6] Cleanup, registry simplification, and v0.4 breaking changes #121 - Phase 5: Cleanup & Deprecation

Branch Status:

• Referenced branch 003-our-aim-to does not exist remotely
• Planning work may be in local/uncommitted state

---

Current Codebase Analysis

Registry Complexity

Examined src/codeweaver/common/registry.py (1,507 lines):

Current Architecture:

• ProviderRegistry: Manages 6 provider types (embedding, sparse, reranking, vector stores, agents, data)
• ModelRegistry: Tracks model capabilities across providers
• ServicesRegistry: Manages service features and dependencies
• Manual instantiation pattern requires:

  registry = get_provider_registry()
  provider = registry.get_embedding_provider()
  instance = registry.get_embedding_provider_instance(provider, singleton=True)

Code Smell Evidence:

# Line 96: TODO comment
# TODO: We need to integrate feature detection into service registration and validation

# Line 11 (indexer.py): TODO comment  
# TODO: register with providers registry

Instantiation Complexity:

• 40+ methods for provider creation/retrieval
• Separate methods for: create_*, get_*_class, get_*_instance
• Singleton management handled manually
• No declarative dependency specification

Related Issues Context

#84 - Register default services (OPEN)

• Currently blocks service registration features
• DI would provide natural registration mechanism

#100 - Improve EmbeddingRegistry (OPEN)

• Proposes configurable size limits and persistence
• DI container would supersede with lifecycle management

#103 - Integrate Indexer with services (OPEN)

• Needs service registry integration
• DI would eliminate need for manual registration

Verdict: All three related issues would be obsoleted or significantly simplified by DI implementation, as claimed.

---

Implementation Difficulty Assessment

Complexity: HIGH

Estimated Effort: 4-6 weeks for Phase 1-2, 12-16 weeks total

Difficulty Factors:

1. Large Surface Area

   • 1,507 lines of registry code
   • 6 provider types × multiple methods
   • Singleton pattern throughout codebase
   • Async/sync mixed usage

2. Type Safety Requirements

   • Project has strict pyright configuration
   • Complex generic types (EmbeddingProvider[Any], VectorStoreProvider[Any])
   • Need proper TypeVar/ParamSpec usage for DI

3. Testing Requirements

   • Constitution mandates evidence-based development
   • Need 100% coverage on DI infrastructure (Phase 1)
   • Integration tests across all provider types

4. Migration Complexity

   • Coexistence period (v0.2-v0.3) requires both patterns
   • Unknown number of instantiation sites in codebase
   • Potential FastMCP integration complications

Risk Assessment

Phase 1 (Foundation): ✅ LOW RISK

• No production changes
• Isolated DI infrastructure
• Clear acceptance criteria

Phase 2 (Core Migration): ⚠️ MEDIUM-HIGH RISK

• Changes core services (Indexer, search)
• Requires careful testing
• Potential for subtle bugs in dependency resolution

Overall Risk: MEDIUM

• Well-structured plan mitigates risk
• Incremental approach allows validation
• BUT: Missing planning docs increase uncertainty

---

Impact Assessment

Positive Impacts

Code Quality:

• 60-70% reduction in dependency boilerplate (claim supported by code comparison)
• Self-documenting dependencies via type annotations
• Better testability via override mechanism
• Eliminates manual singleton management

Developer Experience:

• FastAPI familiarity reduces learning curve
• Cleaner service initialization
• Easier to mock/test

Scalability:

• 40+ providers now, claim of 100+ future
• Mechanical process to add new providers
• Reduces coupling between services

Potential Negative Impacts

Complexity Trade-offs:

• Introduces new abstraction layer (DI container)
• Additional learning curve for contributors unfamiliar with DI
• Potential performance overhead (resolution, caching)

Migration Burden:

• 12-month migration window (v0.2 → v0.4)
• Need to maintain both patterns during transition
• Documentation updates across codebase

Uncertainty Factors:

• Performance impact unknown (claims "within 5%" but no baseline)
• pydantic-ai integration complexity (Phase 3)
• FastMCP compatibility (not validated)

---

Constitutional Compliance Validation

✅ Principle I: AI-First Context

Evidence: Type-annotated dependencies make service relationships explicit, improving code context for AI understanding.

✅ Principle II: Proven Patterns

Evidence: FastAPI's DI is battle-tested with millions of users. Established pattern in pydantic ecosystem.

✅ Principle III: Evidence-Based Development

Status: PARTIALLY COMPLIANT

• ✅ Incremental phases with testing requirements
• ✅ Code examples demonstrate approach
• ❌ Missing planning documents (violates "verifiable evidence" requirement)
• ❌ No performance baseline (claims "within 5%" without data)

Constitutional Concern: The missing planning docs and performance baseline constitute a Red Flag per ARCHITECTURE.md:610-620:

"When APIs behave unexpectedly, files aren't where expected, or code contradicts documentation:

1. Stop current work immediately"

The epic claims extensive planning docs exist, but they don't. This requires clarification before proceeding.

✅ Principle IV: Testing Philosophy

Evidence: Phase 1 requires 100% coverage on DI infrastructure, focuses on integration testing.

✅ Principle V: Simplicity Through Architecture

Evidence: Hides complexity in factories and container, exposes simple interfaces via Depends().

---

Recommendations

1. BEFORE Starting Implementation

Critical Prerequisites:

• Publish planning documents to plans/ directory (or explain their absence)
• Create remaining sub-issues ([DI Phase 2] Core migration - Breaking 75% of circular dependencies #118-[DI Phase 6] Cleanup, registry simplification, and v0.4 breaking changes #121) with detailed acceptance criteria
• Establish performance baseline for current provider instantiation
• Create PoC demonstrating DI with 2-3 provider types

Nice-to-Have:

• Review FastMCP compatibility with DI patterns
• Document FastAPI DI patterns for contributors unfamiliar with framework
• Create migration guide template

2. Suggested Labels

Based on available labels, suggest:

• architecture (already assigned based on scope)
• refactor (this is a refactoring, not new feature)
• providers (core focus on provider system)
• services (affects service registry)
• developer-experience (improves DX significantly)

3. Suggested Enhancements

Epic Description:

• Add "Prerequisites" section explaining planning doc status
• Include performance baseline or note it as Phase 1 deliverable
• Add explicit non-goals (what DI won't solve)

Sub-Issue Creation:

• Use issue templates if available
• Link to epic in each phase issue
• Include rollback criteria for each phase

4. Alternative Approaches to Consider

Option A: Lighter-Weight DI

• Use FastMCP's built-in Context system exclusively
• Avoid custom container implementation
• Trade-off: Less flexibility, simpler implementation

Option B: Hybrid Approach

• Implement DI for new services only
• Leave existing registries for compatibility
• Trade-off: Coexistence period indefinite, but lower risk

Option C: Full FastMCP Alignment

• Wait for FastMCP DI patterns to mature
• Contribute DI features upstream
• Trade-off: Delays improvements, reduces maintenance burden

---

Overall Assessment

Recommendation: APPROVE WITH CONDITIONS

Strengths:

• ✅ Well-structured epic with clear phases
• ✅ Constitutionally compliant approach
• ✅ Addresses real technical debt
• ✅ Good migration strategy

Concerns:

• ⚠️ Missing planning documentation (Red Flag)
• ⚠️ Only 1 of 5 sub-issues created
• ⚠️ No performance baseline established
• ⚠️ High implementation complexity

Conditions for Proceeding:

1. Resolve documentation discrepancy (missing planning docs)
2. Create remaining sub-issues before starting Phase 2
3. Establish performance baseline in Phase 1
4. Consider PoC validation before full commitment

Estimated Impact

If Successful:

• High value - Significant code quality improvement
• High adoption - FastAPI patterns are familiar
• Long-term benefit - Scales to 100+ providers

If Issues Arise:

• Medium risk - Incremental phases allow rollback
• High complexity - Could extend timeline significantly
• Migration burden - 12-month window may not be sufficient

---

Related Issues & Cross-References

Will Obsolete/Simplify:

• Register default services in services registry #84 - Register default services (service registration via DI)
• Improve EmbeddingRegistry: configurable size, dynamic sizing, persistence #100 - Improve EmbeddingRegistry (superseded by DI container)
• Integrate Indexer with services registry and implement persistence loading #103 - Integrate Indexer with services (DI handles integration)

Depends On:

• None identified (self-contained work)

Blocks:

• Future provider additions (current pattern doesn't scale)
• FastMCP advanced patterns (requires cleaner DI)

---

Questions for Issue Author

1. Planning Documentation: Where are the referenced planning docs? Are they in a local branch, different repo, or not yet created?

2. Performance Baseline: What performance characteristics should DI maintain? (The "within 5%" claim needs a baseline)

3. FastMCP Compatibility: Has this approach been validated against FastMCP's architecture? Any concerns about framework compatibility?

4. Timeline: Is the 12-month migration window (v0.2-v0.4) realistic given the codebase size and team capacity?

5. Sub-Issues: What's the timeline for creating issues [DI Phase 2] Core migration - Breaking 75% of circular dependencies #118-[DI Phase 6] Cleanup, registry simplification, and v0.4 breaking changes #121? Should they be created now or as phases complete?

---

Analysis completed: 2025-11-19 21:10 UTC