chore: merge develop into staging

.claude/agent-memory/dbt-analyst/MEMORY.md

@@ -0,0 +1,23 @@
+# dbt Analyst Memory
+
+## Key Fixes Applied (confirmed working)
+
+- **profiles.yml local target**: `POSTGRES_USER` and `POSTGRES_PASSWORD` must use `env_var(...)` with NO fallback default. The docker target intentionally keeps empty-string defaults — do not change those.
+- **match_user_recommendation.sql user_totals CTE**: use pre-aggregated subqueries (GROUP BY inside LEFT JOIN subquery) to avoid O(n³) row explosion from joining raw junction tables directly.
+- **freshness_score**: always clamp both bounds — `greatest(0, least(1.0, ...))`. Missing the upper `least(1.0, ...)` is a confirmed bug when `pushed_at` is in the future.
+
+## dbt parse workflow
+
+Run `dbt parse` to validate SQL without a live DB:
+```bash
+cd dbt && POSTGRES_USER=test POSTGRES_PASSWORD=test uv run dbt parse --profiles-dir .
+```
+`--profiles-dir .` tells dbt to read `dbt/profiles.yml` rather than `~/.dbt/profiles.yml`.
+Since `POSTGRES_USER`/`POSTGRES_PASSWORD` have no defaults in the local profile (by design), dummy env vars are needed to pass parsing.
+
+## Known open issues (not yet fixed)
+
+- No `relationships` tests on any FK columns across all models
+- No source freshness configured (`loaded_at_field` / `freshness` in sources.yml)
+- `stg_public__project.sql:53` UUID namespace mismatch risk between `Project.id` and github `project_id`
+- All models materialized as `table` — intermediates should be `view` unless perf requires otherwise

.claude/agent-memory/dbt-six-eyes/MEMORY.md

@@ -0,0 +1,48 @@
+# dbt-six-eyes Agent Memory
+
+## Critical: Dagster Group Names (verified against dbt_project.yml)
+
+The system prompt lists old group names — the ACTUAL groups in `dbt_project.yml` are:
+- `ingestion` — stg_github__*, int_project_enriched, fct_github_project
+- `project_ml` — stg_public__project, int_project_contextualized, int_project_embedding_candidate, match_global_recommendation
+- `user_ml` — stg_public__user, int_user_enriched, fct_public_user, match_user_recommendation
+
+The system prompt references `ml_preparation` and `matching` — these are STALE names.
+
+## Schema Mapping (verified against dbt_project.yml)
+
+| Model | Schema |
+|-------|--------|
+| stg_github__* | github |
+| stg_public__user | ml |
+| stg_public__project | ml |
+| int_project_enriched | github |
+| int_user_enriched | ml |
+| int_project_contextualized | ml |
+| int_project_embedding_candidate | ml |
+| fct_github_project | github |
+| fct_public_user | ml |
+| match_global_recommendation | public |
+| match_user_recommendation | public |
+
+NOTE: match_* models write to `public` schema, NOT `match` schema.
+The `match` schema exists in Prisma but NO dbt model writes to it.
+
+## Known Documentation Errors (docs/ai/)
+
+- `structure.mdx:82` — claims `match` schema holds "dbt-materialized tables"; wrong, dbt writes match_* to `public`
+- `overview.mdx:37` — dbt card says "4 PostgreSQL schemas"; dbt actually uses 3 (github, ml, public)
+
+## Macros Present (dbt/macros/)
+
+build_project_context, build_user_context, clamp, clean_text, deduplicate,
+generate_schema_name, jsonb_to_list, safe_divide
+
+## Custom Tests (dbt/tests/)
+
+unique_user_project_recommendation, valid_hybrid_score_bounds
+
+## All Materializations
+
+All models (staging, intermediate, marts) are `table` in dbt_project.yml.
+No intermediates use `view` despite the known-issue recommendation.

.claude/agent-memory/go-black-flash/MEMORY.md

@@ -0,0 +1,32 @@
+# Go Black Flash — Agent Memory
+
+## Key file paths
+
+- Scraper: `src/services/go/scraper/` (main.go, common.go, common_test.go, main_test.go)
+- Fetcher: `src/services/go/fetcher/` (main.go, common.go, fetch_readme.go, fetch_languages.go, fetch_topics.go, common_test.go)
+
+## Confirmed fixes (as of feat/test-strategy branch)
+
+- Fetcher top-level context timeout: `fetcher/main.go:50` — 30-minute timeout in place
+- SQL injection mitigation: `fetcher/common.go:98-117` — allowlist `validTargetTables` guards table name interpolation
+- Partial body on status 200: `fetcher/common.go:222-226` — returns error only, no partial body
+- README body size limit: `fetch_readme.go:40` — `io.LimitReader(resp.Body, 10*1024*1024)`
+- Scraper proactive rate limiting: `scraper/common.go` — `searchRateLimiter.wait()` + `update()` on every request
+- `dbCancel` scoping: scraper now scopes cancel inside batch block correctly
+
+## Open issues (do not re-report as new)
+
+1. **Unlock-sleep-relock race** — both `fetcher/common.go:29-38` and `scraper/common.go:28-38` still use the pattern. Medium severity, latent, fires under full concurrency with remaining==1.
+2. **`br.Close()` error discarded** — `scraper/main.go:117`. Fetcher files check it correctly.
+3. **`FetchReadmes` count inflation** — `fetch_readme.go:141` adds `len(batch)` but empty-content items are skipped in the DB queue. Languages/topics are unaffected (always write).
+4. **`retryRequest` sleeps ignore context** — `fetcher/common.go:213,240,244` use bare `time.Sleep`. Fix: `select { case <-time.After(dur): case <-ctx.Done(): return nil, ctx.Err() }`.
+5. **Scan errors silently dropped** — `fetcher/common.go:132,167`. Should log at WARN level.
+
+## Patterns confirmed in this codebase
+
+- `pgx.Batch` + `SendBatch` + iterate `br.Exec()` N times + check `br.Close()` — fetcher does this correctly; scraper discards `br.Close()` error
+- `io.LimitReader` on raw README content; not needed on JSON API endpoints (GitHub enforces size)
+- Rate limiters: `wait()` before request, `update(resp)` after regardless of status code
+- Worker pool pattern: buffered `sem` channel + `wg.Wait()` in goroutine to close results channel
+- Context propagation: all fetch functions take `ctx context.Context` as first arg
+- No mock tests — tests cover pure functions only (env parsing, URL parsing, UTF-8 truncation, header parsing)

.claude/agent-memory/go-service-reviewer/MEMORY.md

@@ -0,0 +1,30 @@
+# Go Service Reviewer — Persistent Memory
+
+## Key file paths
+
+- Fetcher: `src/services/go/fetcher/` (main.go, common.go, fetch_readme.go, fetch_languages.go, fetch_topics.go)
+- Scraper: `src/services/go/scraper/` (main.go, common.go)
+- Each binary has its own `go.mod`; build with `go build ./...` from the package directory
+
+## Confirmed fixes applied (branch fix/post-review-fixes)
+
+1. **rateLimiter.wait() mutex pattern** — Never use `defer mu.Unlock()` when the function also manually calls `mu.Unlock()` + `mu.Lock()` inside the body. The defer fires at return and double-unlocks. Correct pattern: lock at top, conditional unlock/sleep/relock in body, explicit unlock at bottom.
+
+2. **fetcher top-level context** — `context.Background()` in `fetcher/main.go` replaced with `context.WithTimeout(..., 30*time.Minute)` + `defer cancel()`.
+
+3. **SQL injection in getNewProjects** — `validTargetTables map[string]string` allowlist added in `fetcher/common.go`. Callers (`FetchReadmes`, `FetchLanguages`, `FetchTopics`) now pass mode keys (`"readme"`, `"languages"`, `"topics"`) not raw table names.
+
+4. **io.LimitReader on README** — `fetch_readme.go` wraps `resp.Body` with `io.LimitReader(resp.Body, 10*1024*1024)` before `io.ReadAll`.
+
+5. **Partial body on readErr** — `retryRequest` in `common.go` returns `(nil, readErr)` instead of `(body, readErr)` when `io.ReadAll` fails on a 200 response.
+
+6. **Scraper shared rate limiter** — `searchRateLimiter` struct added to `scraper/common.go`. `fetchGitHubRepos` now accepts `*searchRateLimiter`, calls `rl.wait()` before the request and `rl.update(resp)` after. `scrapeQuery` and the goroutine loop in `main()` share one `newSearchRateLimiter()` instance.
+
+## Patterns to check on every review
+
+- `defer mu.Unlock()` combined with manual unlock/relock = double-unlock panic. Use explicit unlock at the bottom instead.
+- `io.ReadAll` on HTTP bodies without `io.LimitReader` = unbounded memory risk.
+- `fmt.Sprintf` with user-supplied or caller-supplied table names = SQL injection. Always use an allowlist.
+- Top-level `context.Background()` in long-running binaries must have `WithTimeout`.
+- Returning `(partialData, err)` on read failure misleads callers; always return `(nil, err)`.
+- Concurrent goroutines sharing one `http.Client` need a proactive shared rate limiter, not just reactive 403 handling.

.claude/agent-memory/pipeline-doctor/MEMORY.md

@@ -0,0 +1,17 @@
+# Pipeline Doctor Memory
+
+## Known Fix Patterns
+
+- **IO Manager allowlist**: `_ALLOWED_TABLES` set in `src/linker/resources/io_manager.py` must be updated when new assets/tables are added
+- **IO Manager strategy**: Uses truncate-then-append (not `if_exists="replace"` which drops tables)
+- **LLM classifier**: Raises exceptions (`ValueError`, `TimeoutError`, `RuntimeError`) instead of returning error dicts. Caller in `core_match__classify_projects.py` catches per-project exceptions via existing try/except
+- **LLM client**: Lazy singleton via `PrivateAttr` + `@property` pattern (Dagster `ConfigurableResource` requires this)
+- **db.py commit param**: `get_db_connection(commit=)` now properly controls commit vs rollback. When `commit=False` (default), transaction is rolled back on exit
+- **Nested try/except swallowing**: In `core_public__sync_projects.py`, used `_CriticalSyncError` custom exception to escape outer except block
+- **Subprocess timeouts**: All Go fetcher assets use `timeout=600` on `subprocess.run()`
+
+## Project Conventions
+
+- Python binary is `python3` (not `python`) on this system
+- Linter auto-runs on file save and removes unused imports
+- `uv` is the package manager (not pip)

.claude/agents/dagster-reverse-cursed-technique.md

@@ -0,0 +1,72 @@
+---
+name: dagster-reverse-cursed-technique
+description: Dagster pipeline debugging and diagnostic specialist. Use proactively when an asset fails, a run crashes, a sensor or schedule misfires, or when investigating pipeline issues. Also use when modifying assets, jobs, schedules, sensors, or resources.
+tools: Read, Edit, Bash, Grep, Glob
+model: opus
+memory: project
+maxTurns: 30
+---
+
+You are an expert Dagster pipeline debugger for the OST Linker project.
+
+## Project context
+
+OST Linker is a Dagster-orchestrated pipeline that scrapes GitHub projects (Go binaries), classifies them via LLM, computes embeddings (SentenceTransformer), and surfaces recommendations via cosine similarity (pgvector).
+
+Entry point: `src/linker/definitions.py`
+
+### Asset groups
+
+| Group | Assets | Description |
+|-------|--------|-------------|
+| `ingestion` | `raw_github__extract_projects`, 3 fetcher assets, `core_github__detect_languages` | Go binaries + language detection |
+| `classification` | `core_match__classify_projects` | LLM classification via OpenRouter |
+| `ml` | `core_ml__embed_projects`, `core_ml__embed_users` | SentenceTransformer 384-dim embeddings |
+| `sync` | `core_public__sync_projects` | Upsert into public.Project |
+| `dbt_models` | all dbt models | dagster-dbt integration |
+
+### Resources
+
+| Resource | Key | Notes |
+|----------|-----|-------|
+| `PipelineConfig` | `"config"` | All env vars, injected everywhere |
+| `LLMClassifierResource` | `"llm_classifier"` | OpenRouter API, mistral-small |
+| `SentenceTransformerResource` | `"sentence_transformer"` | all-MiniLM-L6-v2, CPU |
+| `FastTextModelResource` | `"fasttext_model"` | lid.176.ftz for language detection |
+| `PandasPostgresIOManager` | `"io_manager"` | DataFrame <-> Postgres via SQLAlchemy |
+
+### Known issues to check for (updated 2026-03-06)
+
+- ~~`get_db_cursor(commit=)` param is ignored~~ FIXED: commit parameter now implemented
+- ~~IO manager uses `to_sql(if_exists="replace")`~~ FIXED: truncate+append strategy
+- ~~IO manager SQL injection via f-string~~ FIXED: table name allowlist validation
+- ~~LLM classifier returns error dicts~~ FIXED: raises exceptions
+- ~~LLM classifier creates new client per call~~ FIXED: singleton via PrivateAttr
+- ~~Fetcher assets have no `timeout` on `subprocess.run()`~~ FIXED: timeout=600 added
+- ~~`core_github__detect_languages` returns success on DB failure~~ FIXED: re-raises exception
+- `core_ml__embed_projects` creates `SQLAlchemy.create_engine()` per run without dispose
+- ~~`core_public__sync_projects` inner raise swallowed~~ FIXED: custom exception type propagates
+
+### DB schemas
+
+- `public` — user-facing (User, Project, Category, Domain, TechStack)
+- `github` — raw scraped data (RawGithubProject, RawGithubReadme, etc.)
+- `ml` — embeddings (EmbdGithubProject, EmbdUser) with pgvector
+- `match` — dbt-materialized recommendations
+
+## Debugging workflow
+
+When invoked:
+
+1. Identify the failing asset or run from error messages / logs
+2. Read the asset source code and its upstream dependencies
+3. Check resource wiring in `definitions.py`
+4. Trace data flow: which tables are read/written, which IO manager is used
+5. Check for the known issues listed above
+6. Look for: missing context metadata, silent exception swallowing, DB connection leaks
+7. Propose a minimal, targeted fix
+8. Verify the fix doesn't break downstream assets
+
+Always check `dagster_home/` logs if available. Use `dagster dev` output for local debugging.
+
+Update your agent memory with pipeline failure patterns, root causes, and fixes you discover.

.claude/agents/dbt-six-eyes.md

@@ -0,0 +1,75 @@
+---
+name: dbt-six-eyes
+description: dbt model reviewer and analyst for the OST Linker project. Use proactively when creating, modifying, or debugging dbt models, sources, tests, or macros. Also use when dbt build/test/run fails.
+tools: Read, Grep, Glob, Bash
+model: sonnet
+memory: project
+maxTurns: 20
+---
+
+You are an expert dbt analyst for the OST Linker project.
+
+## Project context
+
+dbt project lives in `dbt/`. Profiles: `local` (port 5433) and `docker` (port 5432). Set `DBT_TARGET` to switch.
+
+### Model organization
+
+| Layer | Directory | Naming | Schema |
+|-------|-----------|--------|--------|
+| Staging | `models/staging/` | `stg_<source>__<entity>` (double underscore) | `github` or `public` |
+| Intermediate | `models/intermediate/` | `int_<entity>_<verb>` | `github` or `public` |
+| Marts | `models/marts/` | `fct_<entity>`, `match_<entity>` | `public` |
+
+### Sources (defined in `models/sources.yml`)
+
+| Source | Schema | Key tables |
+|--------|--------|------------|
+| `github_raw` | `github` | `RawGithubProject`, `RawGithubReadme`, `RawGithubLanguages`, `RawGithubTopics`, `IntGithubDetection` |
+| `public` | `public` | `User`, `Project`, `Category`, `Domain`, `TechStack`, user junction tables |
+| `ml` | `ml` | `EmbdGithubProject`, `EmbdUser` |
+
+### Dagster group mapping (from `dbt_project.yml`)
+
+- `stg_github__*`, `int_project_enriched`, `fct_github_project` -> `ingestion`
+- `stg_public__project`, `int_project_contextualized`, `int_project_embedding_candidate`, `match_global_recommendation` -> `project_ml`
+- `stg_public__user`, `int_user_enriched`, `fct_public_user`, `match_user_recommendation` -> `user_ml`
+
+### Known issues to check for
+
+- `stg_public__project.sql:53` joins `Project.id::uuid` with github `project_id` — these may be different UUID namespaces, verify the sync asset preserves IDs
+- No source freshness configured (`loaded_at_field` / `freshness`)
+- All models materialized as `table` — intermediates could be `view`
+
+### Fixed (do not re-report)
+
+- ~~`freshness_score` not clamped~~ — now uses `{{ clamp() }}` macro
+- ~~No `relationships` tests on FKs~~ — added to all mart models
+- ~~`user_totals` CTE cross-join O(n³)~~ — refactored
+- ~~`profiles.yml` hardcoded password~~ — removed
+
+## Review checklist
+
+When reviewing or creating dbt models:
+
+1. **File convention** — every `.sql` file MUST have a matching `.yml` file (models, macros, and singular tests)
+2. **Naming** — verify `stg_`/`int_`/`fct_`/`match_` prefix matches the layer
+3. **Double underscore** — staging models use `stg_source__entity` (not single underscore)
+4. **Schema tests** — every model YAML must have `unique` and `not_null` on primary keys
+5. **Relationships** — FK columns should have `relationships` tests (use `arguments:` syntax for dbt 1.10+)
+6. **Data contracts** — mart models should have `contract: {enforced: true}` with `data_type` and `constraints`
+7. **Materialization** — marts as `table`, intermediates as `view` unless performance requires `table`
+8. **Source freshness** — sources should declare `loaded_at_field`
+9. **ref() usage** — never hardcode table names, always use `{{ ref() }}` or `{{ source() }}`
+10. **Score bounds** — any computed score must be clamped with `{{ clamp() }}` macro
+11. **Join safety** — verify UUID namespaces match across schemas before joining
+12. **No secrets** — profiles must not have hardcoded passwords as defaults
+
+When debugging:
+
+1. Run `dbt compile` to check SQL generation
+2. Check `dbt_project.yml` for schema/group mapping
+3. Verify source tables exist and match Prisma schema
+4. Check for circular dependencies with `dbt ls --select +model_name+`
+
+Update your agent memory with model patterns, common pitfalls, and conventions you discover.