[Agentic Phase 2] Integrate data providers into pipeline

[bashandbone]

Phase 2: Data Provider Integration

Parent Epic: #123
Depends On: DI Phase 3 (#119) for data source injection
Target: v0.3
Risk Level: Low-Medium

Integrate external data providers (Tavily, DuckDuckGo, etc.) to enrich code search with documentation, web context, and external knowledge.

Goals

• External data source integration
• Context enrichment from docs and web
• Smart data source selection
• Efficient caching and rate limiting

Current State

Scaffolded infrastructure in:

• codeweaver.providers.data - Data provider scaffolding
• Registry integration ready

Implementation Checklist

Data Provider Setup

• Tavily search integration
  • API client configuration
  • Result parsing and normalization
  • Rate limiting and caching
• DuckDuckGo search integration
  • Search API wrapper
  • Result processing
  • Fallback handling
• Additional providers (optional)
  • GitHub API for repository context
  • StackOverflow API
  • Package registry APIs

Pipeline Integration

• Add data provider injection to search flow
• Implement context enrichment hooks
  • Pre-search: External docs lookup
  • Post-search: Result augmentation
  • On-demand: User-triggered enrichment
• Smart provider selection
  • Query-based routing
  • Cost/latency optimization
  • Fallback chains

Caching & Performance

• Implement response caching
  • Cache key generation
  • TTL management
  • Cache invalidation
• Rate limiting per provider
• Batch request optimization
• Async/parallel execution

Testing

• Unit tests for each provider
• Integration tests with search pipeline
• Performance benchmarks
• Cost tracking tests
• Fallback behavior tests

Configuration

• Provider enable/disable flags
• API key management
• Rate limit configuration
• Cache settings
• Provider priority/routing rules

Success Criteria

• External data enriches search results
• Performance impact < 500ms per query
• Graceful degradation when providers unavailable
• Cost tracking and limits working
• Tests passing
• Documentation complete

Example Use Cases

1. Library usage: "How to use FastAPI?" → Tavily finds official docs
2. Error resolution: "Fix ImportError" → Web search finds solutions
3. Best practices: "Python async patterns" → Multiple sources aggregated
4. API details: "Stripe payment API" → Documentation retrieved

Reference

• Scaffolded code: src/codeweaver/providers/data/
• Related: DI Phase 3 ([DI Phase 4] pydantic-ai integration (agents + data sources) #119) for injection infrastructure

[github-actions]

Claude finished Adam Poulemanos (@bashandbone)'s task —— View job

---

Issue Analysis: Data Provider Integration (Phase 2)

Summary

This issue proposes integrating external data providers (Tavily, DuckDuckGo) into CodeWeaver's search pipeline to enrich code search results with documentation, web context, and external knowledge. This is Phase 2 of the Agentic Enhancement Epic (#123).

Current State Analysis

Infrastructure Status: ✅ Scaffolded and Ready

The data provider infrastructure is already implemented and ready for integration:

1. Provider Abstractions (src/codeweaver/providers/tools/__init__.py):

   • get_data_provider() - Factory for provider instantiation
   • load_default_data_providers() - Dynamic provider loading
   • Wraps pydantic-ai common tools (Tavily, DuckDuckGo)

2. Provider Registry (src/codeweaver/providers/capabilities.py):

   • Tavily: Provider.TAVILY with ProviderKind.DATA
   • DuckDuckGo: Provider.DUCKDUCKGO with ProviderKind.DATA
   • Both registered in PROVIDER_CAPABILITIES

3. Configuration System (src/codeweaver/config/settings.py:95-99):

   • DataProviderSettings model already exists
   • Default settings configured:
     • Tavily: disabled by default (requires API key)
     • DuckDuckGo: enabled by default (no API key needed)

4. Dependencies: pydantic-ai>=1.4.0 already includes both providers

Dependency Status: ⚠️ Blocked

Issue #119 (DI Phase 3) is the stated dependency:

• Status: OPEN
• Purpose: Integrate pydantic-ai agents and data sources via DI system
• Impact: This issue cannot be fully implemented until [DI Phase 4] pydantic-ai integration (agents + data sources) #119 provides DI factories for data sources

However, the scaffolding suggests integration work can proceed in parallel with DI implementation.

Related Issues

Parent Epic: #123 - Agentic enhancements (OPEN)
Sibling Issues:

• [Agentic Phase 1] Integrate agents into find_code pipeline #124 - Phase 1: Agent integration in find_code (OPEN)
• [Agentic Phase 3] Internal context agent for orchestrated search/response #126 - Phase 3: Context agent tooling (OPEN)

Similar Work: No other open issues directly address data provider integration or external context enrichment.

Implementation Complexity Assessment

Difficulty: ⭐⭐⭐ (Medium) - 3/5

Rationale:

• ✅ Provider abstractions exist
• ✅ Configuration system ready
• ✅ Dependencies installed
• ⚠️ Pipeline integration points need design
• ⚠️ Requires coordination with DI system ([DI Phase 4] pydantic-ai integration (agents + data sources) #119)
• ⚠️ Caching strategy needs implementation
• ⚠️ Rate limiting per-provider needed
• ⚠️ Cost tracking infrastructure required

Estimated Effort: 1-2 weeks for 1 developer (assuming #119 is complete)

Constitutional Compliance Review

✅ Aligned with Project Constitution (.specify/memory/constitution.md v2.0.1):

1. Principle II (Proven Patterns): ✅ Leverages pydantic-ai common tools ecosystem
2. Principle III (Evidence-Based): ✅ Uses battle-tested Tavily/DuckDuckGo implementations
3. Principle I (AI-First Context): ✅ Enriches agent understanding with external knowledge
4. Principle V (Simplicity): ✅ Clean provider abstraction maintains flat structure

Architectural Compliance: ✅ Follows plugin architecture pattern (ARCHITECTURE.md § Provider Architecture)

Recommendations

1. Clarify Dependency on #119

The issue states "Depends On: DI Phase 3 (#119)" but the existing scaffolding suggests minimal DI coupling. Consider:

Option A: Proceed with direct integration using existing get_data_provider() factory, then refactor for DI in #119
Option B: Wait for #119 completion and use DI from the start

Recommendation: Option A - The provider interface is already DI-compatible. Implement now using factory functions, DI refactor will be straightforward.

2. Define Pipeline Integration Points

The issue mentions three hooks but lacks specifics:

• Pre-search: When to trigger external docs lookup? (query intent detection needed)
• Post-search: How to merge external results with code results? (ranking strategy needed)
• On-demand: User-triggered via tool parameter? (new find_code parameter needed)

Recommendation: Start with post-search augmentation as simplest integration:

# Pseudo-code example
async def find_code(query: str, enrich_external: bool = False):
    code_results = await search_codebase(query)
    if enrich_external and should_use_external(query):
        external_context = await fetch_external_data(query)
        return merge_results(code_results, external_context)
    return code_results

3. Implement Smart Provider Selection

Avoid unnecessary external calls:

• Query classification: "library usage" → Tavily docs, "error fix" → web search
• Cost awareness: Free tier limits, fallback chains (Tavily → DuckDuckGo → none)
• Performance budget: Honor <500ms latency target from success criteria

Recommendation: Use simple heuristics initially:

• Keywords like "docs", "API", "official" → Tavily
• Keywords like "error", "fix", "solution" → DuckDuckGo
• Default: No external enrichment (user must opt-in via parameter)

4. Caching Strategy

External API calls are expensive (cost + latency):

Recommendation: Implement response caching:

• Cache key: hash(provider, query, params)
• TTL: 24 hours for docs, 1 hour for web search
• Storage: Use existing vector store or simple file-based cache
• Invalidation: LRU eviction at 1000 entries

5. Testing Approach

Per Constitution Principle IV (Testing Philosophy):

Integration tests over unit tests:

@pytest.mark.integration
@pytest.mark.external_api
async def test_tavily_enrichment():
    """Test find_code with Tavily enrichment"""
    result = await find_code("FastAPI authentication", enrich_external=True)
    assert result.external_context is not None
    assert "fastapi" in result.external_context.lower()

Recommendation: Mock external APIs by default, use real APIs in CI with @pytest.mark.external_api

6. Suggested Labels

Based on repository labels:

• ✅ enhancement - Already a new feature
• ✅ agents - Part of agentic epic
• ✅ pipeline - Integrates into search pipeline
• ✅ providers - Extends provider system
• ➕ pydantic-ai - Uses pydantic-ai common tools
• ➕ architecture - Significant architectural integration
• ➕ api - May affect find_code tool signature

Potential Risks & Mitigations

Risk	Impact	Mitigation
External API costs exceed budget	High	Implement rate limiting, caching, free tier monitoring
Latency exceeds 500ms target	Medium	Parallel async calls, timeout enforcement, fallback to code-only
Poor result relevance	Medium	Smart provider selection, ranking tuning, user feedback loop
API key management complexity	Low	Use existing pydantic-settings env var pattern
DI refactor breaks integration	Low	Provider interface already DI-compatible

Next Steps

If proceeding with implementation:

1. Week 1: Core provider integration

   • Implement enrich_external parameter in find_code
   • Add smart provider selection heuristics
   • Implement basic caching (file-based)
   • Add rate limiting per provider

2. Week 2: Pipeline integration & testing

   • Integrate into hybrid search pipeline
   • Implement result merging/ranking
   • Add comprehensive integration tests
   • Document configuration options

3. Follow-up: DI migration (after [DI Phase 4] pydantic-ai integration (agents + data sources) #119)

   • Refactor to use DI factories from [DI Phase 4] pydantic-ai integration (agents + data sources) #119
   • Update configuration system
   • Verify no regression in functionality

Constitutional Reminder

Per Constitution § Red Flag Protocol:

When APIs behave unexpectedly, files aren't where expected, or code contradicts documentation:

1. Stop current work immediately
2. Review understanding systematically
3. Never create workarounds without explicit authorization

Good news: Current scaffolding matches documentation. Implementation can proceed confidently.

---

Recommendation: This issue is ready for implementation once #119 provides DI infrastructure, or can proceed immediately using factory pattern with DI refactor deferred.

Branch: claude/issue-125-20251119-2123