Step13: adding requestid middleware and the analyzer endpoint

.agents/scratchpad/2026-02-15-smolterms/request-id-and-timeout-middleware/context.md

@@ -0,0 +1,45 @@
+# Context: Request ID and Timeout Middleware
+
+## Project Structure
+- Module: `github.com/parth/smolterms`
+- Target package: `backend/internal/api/`
+- Key files: `middleware.go`, `router.go`
+- Test files: `middleware_test.go`, `router_test.go`
+
+## Existing Patterns
+
+### Middleware Pattern
+- Simple middleware: `func(http.Handler) http.Handler` (e.g., `CORSMiddleware`)
+- Parameterized middleware: returns `func(http.Handler) http.Handler` (e.g., `LoggingMiddleware(logger)`)
+- Chain order in `NewRouter`: innermost applied first, outermost wraps last
+- Current chain: `mux -> LoggingMiddleware -> CORSMiddleware`
+
+### Testing Pattern
+- Package-level tests (`package api`)
+- `httptest.NewRecorder()` + `httptest.NewRequest()` for unit tests
+- `httptest.NewServer()` for integration tests (router_test.go)
+- Table-driven subtests with `t.Run()`
+- `bytes.Buffer` + `slog.NewTextHandler` for capturing log output
+
+### Response Writer Pattern
+- `statusRecorder` wraps `http.ResponseWriter` to capture status code
+- Write-once guard on `WriteHeader`
+
+## Requirements Summary
+
+1. **Request ID Middleware**: Generate UUID v4, set `X-Request-ID` header, respect incoming header, inject into context
+2. **RequestIDFromContext helper**: Extract request ID from context
+3. **Timeout Middleware**: `context.WithTimeout` on request context, configurable duration
+4. **LoggingMiddleware update**: Include `request_id` in log output
+5. **Router update**: Wire middleware in order: CORS -> RequestID -> Timeout -> Logging -> routes
+
+## Dependencies
+- No `github.com/google/uuid` in go.mod - will use `crypto/rand` to generate UUID v4 (avoids adding dependency)
+- No existing context key patterns in codebase - introduce from scratch
+
+## Design Decisions
+- Use `crypto/rand` for UUID v4 generation (no new dependency needed)
+- Unexported `contextKey` type to prevent collisions
+- `TimeoutMiddleware(duration)` returns `func(http.Handler) http.Handler` (parameterized pattern, matching `LoggingMiddleware`)
+- Do NOT use `http.TimeoutHandler` (known issues with response hijacking)
+- Default timeout: 60 seconds

.agents/scratchpad/2026-02-15-smolterms/request-id-and-timeout-middleware/plan.md

@@ -0,0 +1,63 @@
+# Plan: Request ID and Timeout Middleware
+
+## Test Strategy
+
+### Request ID Middleware Tests
+1. **Generates UUID when no header present** - Request without X-Request-ID -> response has X-Request-ID header with valid UUID format
+2. **Uses existing header value** - Request with X-Request-ID: "test-id-123" -> response X-Request-ID = "test-id-123"
+3. **Injects request ID into context** - Downstream handler calls `RequestIDFromContext(r.Context())` -> returns the ID
+4. **Generated ID is valid UUID v4 format** - Verify format matches `xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx`
+5. **Calls next handler** - Verify the downstream handler is invoked
+
+### Timeout Middleware Tests
+6. **Sets context deadline** - After middleware, `r.Context().Deadline()` returns true with deadline ~60s from now
+7. **Context cancelled after deadline** - With very short timeout (1ms), slow handler sees `context.DeadlineExceeded`
+8. **Calls next handler** - Verify the downstream handler is invoked
+9. **Cancel function called** - Verify no goroutine leak (cancel is deferred)
+
+### Logging Middleware Update Tests
+10. **Includes request_id in log output** - When request_id is in context, log entry contains `request_id=<value>`
+11. **Handles missing request_id** - When no request_id in context, log entry does NOT contain request_id field (or contains empty)
+
+### RequestIDFromContext Tests
+12. **Returns ID when present** - Context with request ID -> returns the ID string
+13. **Returns empty string when absent** - Empty context -> returns ""
+
+### Router Integration Tests
+14. **X-Request-ID header present on health** - Health endpoint response includes X-Request-ID
+15. **Existing router tests still pass** - All 4 existing TestNewRouter subtests pass
+
+## Implementation Plan
+
+### Step 1: Write tests (RED phase)
+- Add tests to `middleware_test.go` for RequestIDMiddleware, TimeoutMiddleware, RequestIDFromContext, and updated LoggingMiddleware
+- Add integration test to `router_test.go` for X-Request-ID header presence
+
+### Step 2: Implement middleware (GREEN phase)
+1. Add context key type and constant to `middleware.go`
+2. Implement `RequestIDMiddleware` in `middleware.go`
+3. Implement `RequestIDFromContext` helper in `middleware.go`
+4. Implement `TimeoutMiddleware` in `middleware.go`
+5. Update `LoggingMiddleware` to include request_id
+6. Update `NewRouter` to wire new middleware
+
+### Step 3: Refactor
+- Review for consistency with existing patterns
+- Ensure no unnecessary complexity
+
+## Implementation Checklist
+- [ ] Tests written for RequestIDMiddleware
+- [ ] Tests written for TimeoutMiddleware
+- [ ] Tests written for RequestIDFromContext
+- [ ] Tests written for LoggingMiddleware update
+- [ ] Tests written for router integration
+- [ ] All tests fail (RED confirmed)
+- [ ] Context key type and RequestIDFromContext implemented
+- [ ] RequestIDMiddleware implemented
+- [ ] TimeoutMiddleware implemented
+- [ ] LoggingMiddleware updated
+- [ ] NewRouter updated
+- [ ] All tests pass (GREEN confirmed)
+- [ ] Refactoring complete
+- [ ] Existing tests still pass
+- [ ] Build succeeds

.agents/scratchpad/2026-02-15-smolterms/request-id-and-timeout-middleware/progress.md

@@ -0,0 +1,37 @@
+# Progress: Request ID and Timeout Middleware
+
+## Setup
+- [x] Documentation directory created
+- [x] Instruction files discovered (none found besides CLAUDE.md)
+- [x] Existing code explored and patterns documented
+
+## Explore
+- [x] Design doc section 6 read (error handling, 504 timeout, context propagation)
+- [x] Existing middleware.go analyzed (CORSMiddleware, LoggingMiddleware, statusRecorder)
+- [x] Existing router.go analyzed (NewRouter with middleware chain)
+- [x] Existing tests analyzed (unit + integration patterns)
+- [x] No existing context key patterns in codebase
+
+## Plan
+- [x] Test strategy designed
+- [x] Implementation plan created
+
+## Code
+- [x] TDD RED phase - tests written and confirmed failing (12 compilation errors)
+- [x] TDD GREEN phase - implementation written and all 27 API tests passing
+- [x] Refactor phase - code reviewed, consistent with existing patterns
+- [x] Validation - all backend tests pass, build succeeds, API package at 100% coverage
+
+### TDD Cycles
+1. RED: Added 13 new test cases across 4 test functions, confirmed compilation failure
+2. GREEN: Implemented contextKey, RequestIDMiddleware, RequestIDFromContext, TimeoutMiddleware, updated LoggingMiddleware, updated NewRouter
+3. REFACTOR: Reviewed naming, documentation, error handling, code organization - all consistent
+
+### Design Decisions
+- Used `crypto/rand` for UUID v4 generation (no new dependency)
+- Unexported `contextKey` type prevents context key collisions
+- `generateUUID` is unexported helper (internal implementation detail)
+- LoggingMiddleware conditionally includes request_id only when present
+
+## Commit
+- [ ] Changes committed

.agents/scratchpad/2026-02-15-smolterms/task-02-analyze-handler-and-routing/context.md

@@ -0,0 +1,31 @@
+# Context: Task 02 - Analyze Handler and Routing
+
+## Project Structure
+- Go 1.22+ with stdlib `net/http` method-based routing
+- Handler factory pattern: `NewXxxHandler(deps) http.HandlerFunc`
+- Response helpers: `WriteJSON[T]` and `WriteError` in `api/response.go`
+- Response types in `backend/internal/types/response.go`
+- White-box testing with `httptest.NewRecorder` + `httptest.NewRequest`
+- Middleware stack: CORS -> RequestID -> Timeout -> Logging -> routes
+
+## Requirements
+1. Create `NewAnalyzeHandler` factory in `handler.go` - accepts `*analyzer.Analyzer` and `*slog.Logger`
+2. Decode JSON body into `analyzer.AnalysisRequest`, validate url/html non-empty
+3. Error mapping: malformed JSON -> 400, empty fields -> 400, deadline exceeded -> 504, other errors -> 502
+4. Update `NewHealthHandler` to accept optional `func(ctx) error` for Qdrant health check
+5. Update `NewRouter` to accept analyzer dependency and register `POST /api/v1/analyze`
+6. Update `main.go` to pass new dependencies
+
+## Existing Patterns
+- Factory functions return `http.HandlerFunc` closures capturing dependencies
+- Nil-safe: handlers default gracefully on nil deps
+- `WriteJSON` / `WriteError` for all responses
+- `got/want` naming in test assertions
+- Table-driven subtests with `t.Run`
+- Handler tests use `httptest.NewRecorder`, router tests use `httptest.NewServer`
+
+## Dependencies
+- `analyzer.Analyzer` has method: `Analyze(ctx, AnalysisRequest) (*AnalysisResult, error)`
+- `analyzer.AnalysisRequest` has `URL string` and `HTML string` with json tags
+- `analyzer.AnalysisResult` is the full response type (already has json tags matching API contract)
+- For testing: need to mock `*analyzer.Analyzer` - cannot construct directly (panics on nil rag/llm)

.agents/scratchpad/2026-02-15-smolterms/task-02-analyze-handler-and-routing/plan.md

@@ -0,0 +1,54 @@
+# Plan: Task 02 - Analyze Handler and Routing
+
+## Test Strategy
+
+### Mocking Approach
+The `Analyzer` struct uses concrete dependencies (rag.Pipeline, llm.LLMClient) that panic on nil. For handler tests, we need to create an interface that `Analyze` satisfies, or use a thin wrapper. The cleanest approach: define a local `pipelineRunner` interface in the handler with `Analyze(ctx, AnalysisRequest) (*AnalysisResult, error)` that `*analyzer.Analyzer` satisfies. This allows easy mocking in tests.
+
+### Analyze Handler Tests (handler_test.go)
+1. **Valid request returns 200 with AnalysisResult** - POST with valid url+html, mock returns result
+2. **Missing URL returns 400** - POST with empty url, expects error message about url
+3. **Missing HTML returns 400** - POST with empty html, expects error message about html
+4. **Malformed JSON returns 400** - POST with invalid JSON body, expects error about invalid JSON
+5. **Empty body returns 400** - POST with no body
+6. **Analyzer timeout returns 504** - Mock returns context.DeadlineExceeded, expects 504
+7. **Analyzer error returns 502** - Mock returns generic error, expects 502
+8. **Non-policy result returns 200** - Mock returns result with RiskNotPolicy, expects 200
+9. **Content-Type is application/json** - All responses should have correct content type
+
+### Health Handler Tests (handler_test.go)
+10. **Qdrant connected** - Health check func returns nil, expects "connected"
+11. **Qdrant unavailable** - Health check func returns error, expects "unavailable"
+12. **Nil health check falls back to URL** - No func provided, shows URL (backward compat)
+
+### Router Tests (router_test.go)
+13. **POST /api/v1/analyze routes to handler** - Integration test with mock analyzer
+14. **GET /api/v1/analyze returns 405** - Wrong method
+15. **OPTIONS /api/v1/analyze returns 204 with CORS** - Preflight support
+
+## Implementation Plan
+
+### Step 1: Define pipeline interface for testability
+Add a `PipelineRunner` interface in handler.go:
+```
+type PipelineRunner interface {
+    Analyze(ctx context.Context, req analyzer.AnalysisRequest) (*analyzer.AnalysisResult, error)
+}
+```
+`*analyzer.Analyzer` implicitly satisfies this.
+
+### Step 2: Implement NewAnalyzeHandler
+Factory: `NewAnalyzeHandler(pipeline PipelineRunner, logger *slog.Logger) http.HandlerFunc`
+- Decode JSON, validate, call Analyze, map errors to status codes
+
+### Step 3: Update NewHealthHandler
+Change signature: `NewHealthHandler(cfg *config.Config, qdrantCheck func(ctx context.Context) error) http.HandlerFunc`
+- If qdrantCheck != nil: call it, report "connected" or "unavailable"
+- If qdrantCheck == nil: fall back to current behavior (show URL)
+
+### Step 4: Update NewRouter
+Change signature: add PipelineRunner parameter
+Register `POST /api/v1/analyze` route
+
+### Step 5: Update main.go
+Pass nil for analyzer and qdrant check for now (full wiring deferred to later steps)

.agents/scratchpad/2026-02-15-smolterms/task-02-analyze-handler-and-routing/progress.md

@@ -0,0 +1,19 @@
+# Progress: Task 02 - Analyze Handler and Routing
+
+## Setup
+- [x] Created documentation directory
+- [x] Discovered instruction files (CLAUDE.md)
+- [x] Explored existing codebase patterns
+- [x] Created context.md
+
+## Implementation Checklist
+- [ ] Design test strategy and plan
+- [ ] Write handler tests (RED phase)
+- [ ] Implement `NewAnalyzeHandler` in handler.go
+- [ ] Update `NewHealthHandler` signature and behavior
+- [ ] Update `NewRouter` to accept analyzer and register analyze route
+- [ ] Update `main.go` for new router signature
+- [ ] Update router tests for analyze endpoint
+- [ ] Run all tests and verify GREEN
+- [ ] Refactor and validate
+- [ ] Commit

backend/README.md

@@ -0,0 +1,5 @@
+## Todo
+
+[] - Make the request timeout an env config option
+[] - Decide how to prevent multiple requests by same user or different users - how to set up limits and prevent DOS
+[] - Testing metrics with some privacy policy datasets to evaluate model and chunking/retrieval strategies

backend/cmd/server/main.go

@@ -18,7 +18,11 @@ func main() {
 
 	logger := config.NewLogger(cfg.LogLevel)
 
-	router := api.NewRouter(logger, cfg)
+	// TODO: Construct analyzer pipeline dependencies (rag.Pipeline, llm.LLMClient,
+	// embedding.EmbeddingClient, vectorstore.VectorStore, cache) and pass
+	// the *analyzer.Analyzer as the pipeline parameter.
+	// TODO: Pass a Qdrant health check function once the vector store is wired.
+	router := api.NewRouter(logger, cfg, nil, nil)
 
 	addr := ":" + cfg.Port
 	logger.Info("starting server", "addr", addr)

backend/internal/api/handler.go

@@ -1,25 +1,103 @@
 package api
 
 import (
+	"context"
+	"encoding/json"
+	"errors"
+	"log/slog"
 	"net/http"
 
+	"github.com/parth/smolterms/backend/internal/analyzer"
 	"github.com/parth/smolterms/backend/internal/config"
 	"github.com/parth/smolterms/backend/internal/types"
 )
 
+// PipelineRunner abstracts the analysis pipeline for testability.
+// *analyzer.Analyzer satisfies this interface.
+type PipelineRunner interface {
+	Analyze(ctx context.Context, req analyzer.AnalysisRequest) (*analyzer.AnalysisResult, error)
+}
+
+// NewAnalyzeHandler returns a handler that accepts HTML content and a URL,
+// runs the analysis pipeline, and returns the result as JSON.
+// If pipeline is nil, the handler returns 503 Service Unavailable.
+// If logger is nil, the default slog logger is used.
+func NewAnalyzeHandler(pipeline PipelineRunner, logger *slog.Logger) http.HandlerFunc {
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return func(w http.ResponseWriter, r *http.Request) {
+		if pipeline == nil {
+			WriteError(w, http.StatusServiceUnavailable, "Analysis service not configured")
+			return
+		}
+
+		var req analyzer.AnalysisRequest
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			WriteError(w, http.StatusBadRequest, "Invalid JSON in request body")
+			return
+		}
+
+		if req.URL == "" {
+			WriteError(w, http.StatusBadRequest, "The url field is required and must be non-empty")
+			return
+		}
+		if req.HTML == "" {
+			WriteError(w, http.StatusBadRequest, "The html field is required and must be non-empty")
+			return
+		}
+
+		requestID := RequestIDFromContext(r.Context())
+
+		result, err := pipeline.Analyze(r.Context(), req)
+		if err != nil {
+			if errors.Is(err, context.DeadlineExceeded) {
+				logger.Error("analysis timed out",
+					"url", req.URL,
+					"request_id", requestID,
+					"error", err,
+				)
+				WriteError(w, http.StatusGatewayTimeout, "Analysis timed out. Please try again.")
+				return
+			}
+
+			logger.Error("analysis failed",
+				"url", req.URL,
+				"request_id", requestID,
+				"error", err,
+			)
+			WriteError(w, http.StatusBadGateway, "Analysis service temporarily unavailable. Please try again.")
+			return
+		}
+
+		WriteJSON(w, http.StatusOK, result)
+	}
+}
+
 // NewHealthHandler returns a handler that reports the health status of the server
 // and its configured services. cfg may be nil (services will show as not_configured).
-func NewHealthHandler(cfg *config.Config) http.HandlerFunc {
+// qdrantCheck is an optional function that checks Qdrant connectivity. If nil,
+// the handler falls back to showing the configured Qdrant URL.
+func NewHealthHandler(cfg *config.Config, qdrantCheck func(ctx context.Context) error) http.HandlerFunc {
 	if cfg == nil {
 		cfg = &config.Config{}
 	}
 	return func(w http.ResponseWriter, r *http.Request) {
-		qdrant := cfg.QdrantURL
-		if qdrant == "" {
-			qdrant = "localhost:6334"
+		qdrantStatus := cfg.QdrantURL
+		if qdrantStatus == "" {
+			qdrantStatus = "localhost:6334"
 		}
+
+		if qdrantCheck != nil {
+			if err := qdrantCheck(r.Context()); err != nil {
+				qdrantStatus = "unavailable"
+			} else {
+				qdrantStatus = "connected"
+			}
+		}
+
 		services := map[string]string{
-			"qdrant":    qdrant,
+			"qdrant":    qdrantStatus,
 			"anthropic": apiKeyStatus(cfg.AnthropicAPIKey),
 			"openai":    apiKeyStatus(cfg.OpenAIAPIKey),
 		}