polishing

.gitignore

@@ -46,3 +46,4 @@ next-env.d.ts
 
 # logs
 logs/
+/requests/http-client.private.env.json

AI_ONBOARDING.md

@@ -5,10 +5,10 @@ Goal: Make minimal, correct changes that improve the app while preserving OpenAI
 
 1) Project Snapshot
 - Name: ChatForge (full‑stack AI chat)
-- Frontend: Next.js + React (TypeScript)
-- Backend: Node.js (Express, ESM) acting as an OpenAI‑compatible proxy
-- Streaming: End‑to‑end SSE for chat responses
-- Status: MVP complete; testing infrastructure in place; conversation persistence in development
+- Frontend: Next.js 15 + React 19 (TypeScript) with enhanced UI components
+- Backend: Node.js (Express, ESM) acting as an OpenAI‑compatible proxy with tool orchestration
+- Streaming: End‑to‑end SSE for chat responses with tool events and thinking support
+- Status: MVP complete; tool orchestration system complete; testing infrastructure in place; conversation persistence in development
 
 2) Core Principles
 - Keep diffs small, focused, and documented.
@@ -18,8 +18,11 @@ Goal: Make minimal, correct changes that improve the app while preserving OpenAI
 - Update docs when changing behavior (README.md, docs/*).
 
 3) Repository Map
-- frontend/: Next.js app (app/, components/, lib/)
+- frontend/: Next.js app (app/, components/, lib/, hooks/, contexts/)
 - backend/: Express proxy (src/routes/, src/lib/, src/db/)
+  - src/lib/tools.js: Server-side tool registry and execution
+  - src/lib/unifiedToolOrchestrator.js: Unified tool orchestration system
+  - src/lib/iterativeOrchestrator.js: Iterative workflows with thinking support
 - docs/: Overview/specs/progress/security
 - docker-compose*.yml, dev.sh: Dev orchestration
 
@@ -31,7 +34,7 @@ Option B: Docker Production
 - docker compose -f docker-compose.yml up --build (frontend on 3000)
 Option C: Docker Development (with hot reload)
 - docker compose -f docker-compose.dev.yml up --build (frontend on 3000)
-Note: Dev compose includes hot reload and development dependencies.
+Note: Dev compose includes hot reload and development dependencies with Turbopack for faster iteration.
 
 5) Environment/Secrets
 - backend/.env requires OPENAI_API_KEY (or provider‑compatible key)
@@ -40,55 +43,77 @@ Note: Dev compose includes hot reload and development dependencies.
 6) API Contract (must preserve)
 - POST /v1/responses → primary endpoint with conversation continuity support
 - POST /v1/chat/completions → OpenAI‑compatible endpoint for compatibility
-- Supports text/event-stream (SSE) for streaming tokens
+- Supports text/event-stream (SSE) for streaming tokens and tool events
 - Backend injects Authorization header from server env
 - Do not break request/response JSON shape or streaming semantics
 - Responses API includes `previous_response_id` for conversation linking
+- Tool support: tools array enables server-side tool execution with iterative workflows
+- Research mode: `research_mode: true` enables multi-step tool orchestration with thinking
 
 7) Streaming Expectations
 - Frontend consumes SSE and renders partial chunks progressively
 - Backend must flush tokens promptly; no buffering of full responses
 - Abort support: requests should be cancellable
+- Tool events: streaming includes tool_calls, tool_output events for real-time feedback
+- Thinking support: iterative orchestration streams AI reasoning between tool calls
 
 8) Rate Limiting & Safety
 - In‑memory per‑IP rate limit in backend (keep or improve without regressions)
 - Avoid noisy logs and PII; follow docs/SECURITY.md guidance
 
-9) Coding Standards
+9) Tool Orchestration System (Major Feature)
+- **Server-side tools**: Available tools defined in backend/src/lib/tools.js (get_time, web_search)
+- **Unified orchestrator**: unifiedToolOrchestrator.js automatically adapts streaming/non-streaming
+- **Iterative mode**: iterativeOrchestrator.js supports thinking between tool calls (up to 10 iterations)
+- **Tool execution**: Tools execute server-side with proper error handling and timeouts
+- **Streaming events**: Real-time tool_calls and tool_output events for UI feedback
+- **Research mode**: When enabled, AI can use tools multiple times with reasoning between calls
+- **Tool adding**: Add new tools with Zod validation schemas; they're automatically available
+- **Persistence integration**: Tool results are properly stored in conversation history
+
+10) Coding Standards
 - Use TypeScript/ESM defaults already present
 - Follow existing ESLint/Prettier configuration (backend and frontend configured)
 - Run linting: `npm --prefix backend run lint` and `npm --prefix frontend run lint`
 - Prefer small pure functions; handle errors and edge cases explicitly
 - Maintain strong typing at API boundaries
+- Tool development: Add tools to backend/src/lib/tools.js with proper validation schemas
 
-10) Tests
+11) Tests
 - Comprehensive Jest testing infrastructure for both backend and frontend
 - Tests located under package‑local __tests__/ directories
 - Run tests: `npm --prefix backend test` and `npm --prefix frontend test`
 - Ensure existing behavior remains green; all tests must pass
+- Tool orchestration tests: iterative_orchestration.test.js, unified_tool_system.test.ts
+- Frontend integration tests for enhanced UI components and chat state management
 
-11) Performance & UX
+12) Performance & UX
 - Preserve fast first token time; avoid unnecessary awaits in hot paths
-- Keep UI responsive during streams; don’t block the main thread
+- Keep UI responsive during streams; don't block the main thread
+- Tool orchestration: up to 10 iterations with smart timeout management (30s per request)
+- Quality controls: UI includes quality slider (quick/balanced/thorough) for response control
+- Enhanced components: floating UI positioning with @floating-ui/react for dropdowns
 
-12) Making Changes
+13) Making Changes
 - Seek the smallest viable fix; avoid broad API surface changes
 - If API surface must change, keep OpenAI compatibility and update docs
 - Add comments near non‑obvious logic; update README/docs links as needed
 
-13) Useful Docs
+14) Useful Docs
 - docs/OVERVIEW.md (architecture with current tech stack)
-- docs/API-SPECS.md (both Responses API and Chat Completions API)
-- docs/CONVERSATIONS-SPEC.md (conversation persistence specification)
-- docs/PROGRESS.md (development progress and completed features)
-- docs/TECH-STACK.md (current dependencies and infrastructure)
+- docs/API-SPECS.md (both Responses API and Chat Completions API with tool support)
+- docs/PROGRESS.md (development progress and completed features including tool orchestration)
+- docs/TECH-STACK.md (current dependencies and infrastructure including Next.js 15, React 19)
 - docs/SECURITY.md (security considerations and environment setup)
-- README.md (quick start, build, and testing)
+- README.md (quick start, build, testing, and tool development)
+- backend/src/lib/tools.js (server-side tool registry and examples)
 
-14) Definition of Done (for AI agents)
+15) Definition of Done (for AI agents)
 - Requirement satisfied with minimal diff
-- Streaming and API compatibility intact
+- Streaming and API compatibility intact (including tool events)
 - No secrets leaked; local/dev still runs per README
 - Relevant docs updated when behavior changes
+- Tool orchestration behavior preserved when modifying tool-related code
+- Enhanced UI components maintain accessibility and responsive design
 
 Welcome aboard. Optimize for correctness, compatibility, and small, reviewable changes.

backend/.env.example

@@ -1,5 +1,15 @@
+## Provider selection (default: openai)
+PROVIDER=openai
+
+## Generic provider config (falls back to OpenAI values)
+# PROVIDER_BASE_URL=
+# PROVIDER_API_KEY=
+# PROVIDER_HEADERS_JSON={"X-Custom":"Value"}
+
+## OpenAI-compatible defaults (kept for backward-compat)
 OPENAI_BASE_URL=https://api.openai.com/v1
 OPENAI_API_KEY=sk-xxxxx
+
 DEFAULT_MODEL=gpt-4.1-mini
 TITLE_MODEL=gpt-4.1-mini
 PORT=3001

backend/Dockerfile

@@ -5,11 +5,13 @@ FROM node:20-slim AS dev
 WORKDIR /app
 ENV NODE_ENV=development
 COPY package*.json ./
-RUN npm install
 # Copy source for dev (mounted again via volume in compose)
 COPY src ./src
 COPY .env.example ./
+COPY entrypoint.sh ./
+RUN chmod +x entrypoint.sh
 EXPOSE 3001
+ENTRYPOINT ["./entrypoint.sh"]
 CMD ["npm", "run", "dev"]
 
 # --- Prod stage: lean runtime image (default/final) ---

backend/README.md

@@ -1,17 +1,17 @@
 # Backend
 
-Express-based proxy for OpenAI-compatible chat completions.
+Express-based proxy for OpenAI-compatible chat completions, with pluggable providers.
 
 ## Endpoints
 
-- `POST /v1/chat/completions` – proxies to `OPENAI_BASE_URL/chat/completions` (supports streaming)
+- `POST /v1/chat/completions` – proxies to `${PROVIDER_BASE_URL||OPENAI_BASE_URL}/v1/chat/completions` (supports streaming)
 - `POST /v1/conversations` – create a conversation (feature-flagged)
 - `GET /v1/conversations/:id` – fetch conversation metadata (feature-flagged)
 - `GET /healthz` – health/status info
 
 ## Env Vars (.env)
 
-See `.env.example` for required variables.
+See `.env.example` for required variables. You can select a provider via `PROVIDER` (default: `openai`). Generic keys `PROVIDER_BASE_URL`, `PROVIDER_API_KEY`, and optional `PROVIDER_HEADERS_JSON` are supported; OpenAI-specific vars remain for backward compatibility.
 
 Additional (Sprint 1):
 
@@ -50,7 +50,7 @@ This reduces database write load and avoids timer-based flushes while preserving
 1. Create env file (not copied into image):
    ```bash
    cp .env.example .env
-   # edit OPENAI_API_KEY etc.
+   # edit PROVIDER/OPENAI variables as needed
    ```
 2. Build & run (from repo root):
    ```bash

backend/entrypoint.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+set -e
+
+# Install dependencies
+echo "Installing npm dependencies..."
+npm install
+
+# Execute the original command
+exec "$@"

backend/package.json

@@ -11,9 +11,11 @@
     "start": "NODE_ENV=production node src/index.js",
     "test": "NODE_OPTIONS=--experimental-vm-modules jest",
     "lint": "eslint .",
-    "format": "prettier --write ."
+    "format": "prettier --write .",
+    "migrate": "node scripts/migrate.js"
   },
   "dependencies": {
+    "@blackglory/better-sqlite3-migrations": "^0.1.20",
     "better-sqlite3": "^9.4.3",
     "cors": "^2.8.5",
     "dotenv": "^16.4.5",