TrendReef

TrendReef is a multi-tenant SaaS backend for AI content intelligence and autonomous market research.

Documentation

• docs/README.md - full documentation index
• docs/ARCHITECTURE.md - platform architecture and runtime flow
• docs/API_REFERENCE.md - API contracts
• docs/OPERATIONS.md - runbook and troubleshooting
• docs/FILE_GUIDE.md - file-by-file ownership map
• docs/AGENT_ONBOARDING.md - coding-agent handoff guide
• docs/architecture.mmd - system architecture diagram (Mermaid)
• docs/research-sequence.mmd - research request sequence diagram
• docs/workflow-sequence.mmd - workflow execution sequence diagram

Architecture Summary

• FastAPI async API with tenant-scoped endpoints.
• Multi-provider LLM layer:
  • BaseLLMProvider
  • OllamaProvider
  • OpenAIProvider
  • AnthropicProvider
  • AzureProvider
• ModelRouter supports per-tenant/per-agent/per-task routing.
• LLMService enforces:
  • fallback provider logic
  • circuit breaker
  • rate limits
  • token/latency/cost usage logging
• LangGraph orchestration with agents:
  • SourceAgent
  • WebResearchAgent (TinyFish SSE)
  • IntelligenceAgent
  • StrategyAgent
  • ContentGeneratorAgent
  • PublisherAgent (stub)
  • AnalyticsAgent
• PostgreSQL + pgvector + SQLAlchemy async models.
• Celery worker for async workflow execution.

Project Structure

app/
  agents/
  api/
    routes/
  core/
  models/
  providers/
  repositories/
  services/
  workflows/
workers/
frontend/
  src/
Dockerfile
docker-compose.yml
requirements.txt
.env.example

Database Tables

• tenants
• users
• sources
• raw_posts
• processed_trends
• embeddings
• generated_content
• analytics_events
• external_research_jobs
• model_providers
• model_configs
• tenant_model_preferences
• model_usage_logs

pgvector extension is enabled at startup.

API Endpoints

Base prefix: /api/v1

• GET /health
• GET /provider-health
• GET|POST /sources
• GET /trends
• POST /generate
• GET|POST /analytics
• POST /run-workflow
• GET|POST /research

All business endpoints require header: X-Tenant-ID.

Crawler Integration

TinyFishAutomationClient calls:

POST https://agent.tinyfish.ai/v1/automation/run-sse

Features implemented:

• SSE stream parsing (data: lines)
• partial chunk buffering
• safe JSON parse fallback
• timeout + retries
• structured logging
• graceful errors

WebResearchAgent now calls a crawler abstraction with routing/fallback:

• tinyfish provider
• playwright provider (local browser crawl)

Provider selection:

• request-level via /research payload provider_preference: auto | tinyfish | playwright
• auto tries TinyFish first, then Playwright fallback
• tenant feature flags can disable providers (tinyfish_enabled, playwright_enabled)

WebResearchAgent stores lifecycle and results in external_research_jobs.

For Playwright local crawling, install Chromium once in runtime environment:

playwright install chromium

Docker builds already install Chromium for Playwright provider. If Chromium runtime libs are missing on your host/container, keep provider as auto or tinyfish.

Local Development

1. Copy env:

cp .env.example .env

2. Start stack:

docker compose up --build

3. Frontend:

http://localhost:5173

4. API:

http://localhost:8000/docs

Migrations (Alembic)

Apply migrations:

alembic upgrade head

Recommended with this compose setup (no DB host-port mapping): run Alembic inside the API container so db resolves:

docker compose run --rm api alembic upgrade head

If you run Alembic locally from your host shell, expose DB port yourself and point DATABASE_URL to that host port.

Rollback one revision:

alembic downgrade -1

Seed Defaults

Seed a demo tenant, providers, model configs, and default routing preference:

python -m scripts.seed_defaults

With compose networking, run:

docker compose run --rm api python -m scripts.seed_defaults

This creates tenant slug demo.

Worker

Celery worker runs as worker service in docker-compose.yml.

Manual run:

celery -A app.workers.celery_app.celery_app worker --loglevel=info

Routing and Provider Config

Provider secrets/config are resolved in this order:

1. Tenant DB config (model_providers)
2. .env fallback

Routing preferences are read from tenant_model_preferences by:

• tenant_id
• agent_name
• task_type

If missing, router defaults to:

• primary: ollama
• secondary: openai

Observability

Structured logs capture:

• model latency
• token usage
• cost estimates
• TinyFish job duration
• trend scoring breakdown

Provider status is exposed via /api/v1/provider-health.

Phase-Based Delivery

Phase 1:

• Core app structure, config, logging, DB models.

Phase 2:

• Provider abstraction, router, LLM service resilience.

Phase 3:

• TinyFish SSE research integration + external jobs tracking.

Phase 4:

• LangGraph multi-agent orchestration and workflow endpoint.

Phase 5:

• Dockerization, worker integration, docs and ops ergonomics.

Notes

• Publisher integrations are intentionally plugin-ready stubs.
• Embedding generation is currently placeholder logic and should be connected to a provider embedding API for production rollout.
• Initial Alembic migration is included in migrations/versions/0001_initial_trendreef_schema.py.

Tests

Run tests:

pytest -q

Coverage focus in current suite:

• core resilience primitives (circuit breaker)
• TinyFish SSE line parsing safety
• intelligence scoring + dedupe logic
• API smoke + tenant header enforcement