feat(data): require sourced records across loaders

Author: Seungpyo1007

README.md

@@ -3,73 +3,73 @@
 > **Validation, ingestion, and serving engine for the [TechAPI](https://github.com/GetTechAPI/TechAPI) dataset.**
 
 [![test](https://github.com/GetTechAPI/TechEngine/actions/workflows/test.yml/badge.svg)](https://github.com/GetTechAPI/TechEngine/actions/workflows/test.yml)
- Code: **MIT** · Data: lives in **[TechAPI](https://github.com/GetTechAPI/TechAPI)** (CC-BY-SA 4.0)
+ Code: **MIT** - Data: lives in **[TechAPI](https://github.com/GetTechAPI/TechAPI)** (CC-BY-SA 4.0)
 
 TechEngine owns everything *around* the data: schema validation, the FastAPI
-read API, the static JSON dump generator, the engine's own landing site, and
-(next up) automated coverage checks and a weekly ingestion crawler.
+read API, the static JSON dump generator, the engine's own landing site,
+automated coverage checks, and the weekly ingestion/refresh crawlers.
 
 The dataset and the public-facing playground site live in
 [TechAPI](https://github.com/GetTechAPI/TechAPI) so each can be versioned,
 mirrored, and licensed independently. The site shipped in this repo is the
-engine's own landing — what TechEngine is, what it runs, link out to docs.
+engine's own landing: what TechEngine is, what it runs, and links out to docs.
 
 ## Layout
 
-```
+```text
 app/
-  ├ validate.py        # schema/range/uniqueness checks
-  ├ seed.py            # data/ → SQLModel database
-  ├ dump.py            # API → static JSON tree
-  ├ main.py            # FastAPI entrypoint
-  ├ models/            # SQLModel tables
-  ├ routers/           # /v1/{brands,socs,smartphones,gpus,cpus}
-  ├ schemas/           # Pydantic response models
-  ├ services/          # scoring (algorithm_version-tagged)
-  ├ coverage/          # upstream-vs-curated diff + Markdown report
-  └ ingest/            # draft new records from upstream pages
+  validate.py          # schema/range/uniqueness checks
+  seed.py              # data/ -> SQLModel database
+  dump.py              # API -> static JSON tree
+  main.py              # FastAPI entrypoint
+  models/              # SQLModel tables
+  routers/             # /v1/{brands,socs,smartphones,gpus,cpus}
+  schemas/             # Pydantic response models
+  services/            # scoring (algorithm_version-tagged)
+  coverage/            # upstream-vs-curated diff + Markdown report
+  ingest/              # draft new records from upstream pages
 tests/                 # unit + integration
 site/                  # Astro engine landing (deploys to Pages)
 docs/                  # SPEC / DATA_PIPELINE / DEVELOPMENT
-TechAPI/               # submodule → GetTechAPI/TechAPI (clickable @ <sha> link)
+TechAPI/               # submodule -> GetTechAPI/TechAPI (clickable @ <sha> link)
 .github/workflows/
-  ├ validate-data.yml   # workflow_call: PR-time data validation for TechAPI
-  ├ weekly-refresh.yml  # cron: live-scrape → integrity gate → dump → PR to TechAPI
-  ├ weekly-ingest.yml   # cron: draft new SKUs, open PR against TechAPI
-  ├ coverage-report.yml # cron: gap report, sticky issue (TechEngine + TechAPI)
-  ├ refresh-data.yml    # smoke-test: rebuild the dump on engine (app/**) changes
-  ├ notify-techapi.yml  # push→main: ping TechAPI to bump its TechEngine submodule
-  ├ bump-techapi.yml    # dispatch: advance this repo's TechAPI submodule pointer
-  ├ deploy-pages.yml    # build & deploy engine site + dump
-  └ test.yml            # lint + type-check + tests
+  validate-data.yml    # workflow_call: PR-time data validation for TechAPI
+  weekly-refresh.yml   # cron: live-scrape -> integrity gate -> dump -> PR to TechAPI
+  weekly-ingest.yml    # cron: draft new SKUs, open PR against TechAPI
+  coverage-report.yml  # cron: gap report, sticky issue (TechEngine + TechAPI)
+  refresh-data.yml     # smoke-test: rebuild the dump on engine (app/**) changes
+  notify-techapi.yml   # push to main: ping TechAPI to bump its TechEngine submodule
+  bump-techapi.yml     # dispatch: advance this repo's TechAPI submodule pointer
+  deploy-pages.yml     # build & deploy engine site + dump
+  test.yml             # lint + type-check + tests
 ```
 
-## How the two repos connect
+## How The Two Repos Connect
 
 Both repos live in the **GetTechAPI** org and each includes the other as a git
 **submodule** (a clickable `@ <sha>` pin). Three automations keep them in step:
 
-- **validate-data.yml** (`workflow_call`) — TechAPI's PR-time check calls into
+- **validate-data.yml** (`workflow_call`) - TechAPI's PR-time check calls into
   TechEngine to validate its data.
-- **weekly-refresh.yml** — live-scrapes benchmarks, runs the full-dataset
+- **weekly-refresh.yml** - live-scrapes benchmarks, runs the full-dataset
   integrity gate (`app.validate` + `integrity_check.py --strict`), regenerates
   the static dump, and opens a dated refresh PR against TechAPI.
-- **Submodule autosync** — every push to TechEngine `main` fires
+- **Submodule autosync** - every push to TechEngine `main` fires
   `notify-techapi.yml`, which pings TechAPI to bump its TechEngine pointer;
   conversely `bump-techapi.yml` advances TechEngine's TechAPI pointer when
   TechAPI changes. Bumps are loop-guarded, so each real change converges to one.
 
 Every Python entry point reads data from a **TechAPI checkout**. The location
-can be overridden via `TECHAPI_DATA_DIR`; the default looks for `../TechAPI/data`
-next to this repo, which matches a local dev layout.
+can be overridden via `TECHAPI_DATA_DIR`; by default TechEngine supports both
+layouts: `TechAPI/TechEngine` as a submodule and `TechAPI` beside `TechEngine`.
 
 ## Quickstart
 
 ```bash
 git clone https://github.com/GetTechAPI/TechAPI.git ../TechAPI   # data source
 pip install -e ".[dev]"
 python -m app.validate          # check data integrity
-python -m app.seed              # data/ → ./techapi.db (SQLite)
+python -m app.seed              # data/ -> ./techapi.db (SQLite)
 uvicorn app.main:app --reload   # serve; curl localhost:8000/v1/cpus/ryzen-9-9950x3d
 python -m app.dump              # generate ./dump/v1/... static tree
 ```
@@ -87,14 +87,14 @@ Spins up Postgres 16, seeds from the mounted TechAPI checkout, serves on `:8000`
 ## Roadmap
 
 - [x] Split out from TechAPI; sibling-checkout data pipeline
-- [x] **Coverage gap detector** — diff curated dataset vs upstream catalogs
+- [x] **Coverage gap detector** - diff curated dataset vs upstream catalogs
   and surface missing SKUs as a sticky weekly issue
   ([#1](https://github.com/GetTechAPI/TechEngine/issues/1))
-- [x] **Weekly ingestion crawler** — scrape canonical sources and open PRs
+- [x] **Weekly ingestion crawler** - scrape canonical sources and open PRs
   against TechAPI with new SKUs (requires the `TECHAPI_TOKEN` secret to push)
   ([#2](https://github.com/GetTechAPI/TechEngine/issues/2))
-- [x] **Weekly refresh pipeline** — live benchmark enrichment → full-dataset
-  integrity gate → static dump → dated refresh PR (`weekly-refresh.yml`)
+- [x] **Weekly refresh pipeline** - live benchmark enrichment -> full-dataset
+  integrity gate -> static dump -> dated refresh PR (`weekly-refresh.yml`)
 - [x] **Bidirectional submodule autosync** between TechEngine and TechAPI
 - [ ] More sources (Intel ARK, AMD product pages, TechPowerUp DB)
 

app/coverage/curated.py

@@ -1,20 +1,19 @@
 """Read curated TechAPI slugs from disk.
 
 Resolves the dataset location the same way as ``app.validate`` / ``app.seed``:
-``TECHAPI_DATA_DIR`` env var, falling back to ``../TechAPI/data`` next to this
-repo.
+``TECHAPI_DATA_DIR`` env var, falling back to the local TechAPI checkout.
 """
 
 from __future__ import annotations
 
 import json
-import os
 from pathlib import Path
 
+from app.data_root import get_data_root
+
 
 def data_dir() -> Path:
-    default = Path(__file__).resolve().parent.parent.parent.parent / "TechAPI" / "data"
-    return Path(os.environ.get("TECHAPI_DATA_DIR", default))
+    return get_data_root()
 
 
 def curated_slugs(category: str, manufacturer: str | None = None) -> set[str]:

app/data_root.py

@@ -0,0 +1,22 @@
+"""Resolve the TechAPI data directory for sibling and submodule layouts."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+
+def get_data_root() -> Path:
+    explicit = os.environ.get("TECHAPI_DATA_DIR")
+    if explicit:
+        return Path(explicit)
+
+    repo_parent = Path(__file__).resolve().parent.parent.parent
+    candidates = (
+        repo_parent / "data",
+        repo_parent / "TechAPI" / "data",
+    )
+    for candidate in candidates:
+        if candidate.exists():
+            return candidate
+    return candidates[-1]

app/ingest/__main__.py

@@ -13,11 +13,12 @@
 from __future__ import annotations
 
 import argparse
-import os
 import sys
 from collections.abc import Iterator
 from pathlib import Path
 
+from app.data_root import get_data_root
+
 from .pipeline import run
 from .sources.base import IngestCandidate, IngestSource
 from .sources.wikipedia_cpu import WikipediaCpuIngest
@@ -32,10 +33,7 @@
 
 
 def _default_data_root() -> Path:
-    explicit = os.environ.get("TECHAPI_DATA_DIR")
-    if explicit:
-        return Path(explicit)
-    return Path(__file__).resolve().parent.parent.parent.parent / "TechAPI" / "data"
+    return get_data_root()
 
 
 def _collect(category: str, limit: int | None) -> Iterator[IngestCandidate]:

app/ingest/enrich.py

@@ -28,7 +28,6 @@
 
 import argparse
 import json
-import os
 import sys
 import time
 from collections.abc import Callable
@@ -38,6 +37,8 @@
 
 import httpx
 
+from app.data_root import get_data_root
+
 from .sources import (
     blender,
     cgdirector,
@@ -107,10 +108,7 @@ def markdown_summary(self, source: str = "") -> str:
 
 
 def _default_data_root() -> Path:
-    explicit = os.environ.get("TECHAPI_DATA_DIR")
-    if explicit:
-        return Path(explicit)
-    return Path(__file__).resolve().parent.parent.parent.parent / "TechAPI" / "data"
+    return get_data_root()
 
 
 def _candidates(cpu_root: Path, manufacturer: str | None) -> list[Path]:

app/models/brand.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+from sqlalchemy import JSON, Column
 from sqlmodel import Field, SQLModel
 
 
@@ -17,5 +18,6 @@ class Brand(SQLModel, table=True):
     founded_year: int | None = None
     logo_url: str | None = None
     website: str | None = None
+    source_urls: list[str] = Field(default_factory=list, sa_column=Column(JSON))
     description_en: str | None = None
     description_ko: str | None = None

app/schemas/brand.py

@@ -25,6 +25,7 @@ class BrandRead(BaseModel):
     founded_year: int | None = None
     logo_url: str | None = None
     website: str | None = None
+    source_urls: list[str]
     description_en: str | None = None
     description_ko: str | None = None
     url: str

app/schemas/serializers.py

@@ -49,6 +49,7 @@ def brand_read(brand: Brand) -> BrandRead:
         founded_year=brand.founded_year,
         logo_url=brand.logo_url,
         website=brand.website,
+        source_urls=brand.source_urls,
         description_en=brand.description_en,
         description_ko=brand.description_ko,
         url=url_for("brands", brand.slug),

app/seed.py

@@ -16,22 +16,21 @@
 from __future__ import annotations
 
 import json
-import os
 from datetime import date
 from pathlib import Path
 from typing import Any
 
 from sqlmodel import Session, SQLModel, select
 
+from app.data_root import get_data_root
 from app.database import create_db_and_tables, engine
 from app.models.brand import Brand
 from app.models.cpu import CPU
 from app.models.gpu import DiscreteGPU
 from app.models.smartphone import Smartphone
 from app.models.soc import SoC
 
-_DEFAULT_DATA_DIR = Path(__file__).resolve().parent.parent.parent / "TechAPI" / "data"
-DATA_DIR = Path(os.environ.get("TECHAPI_DATA_DIR", _DEFAULT_DATA_DIR))
+DATA_DIR = get_data_root()
 
 
 def _load_dir(subdir: Path) -> list[dict[str, Any]]:

app/validate.py

@@ -4,26 +4,25 @@
 and foreign-key integrity by slug. Run with ``python -m app.validate``;
 exits non-zero on the first failure set (used by CI ``validate-data.yml``).
 
-``DATA_DIR`` defaults to ``../TechAPI/data`` (sibling checkout) but can be
-overridden via the ``TECHAPI_DATA_DIR`` environment variable when running
-inside CI where TechAPI is checked out to a different path.
+``DATA_DIR`` defaults to the nearest TechAPI data checkout and can be
+overridden via the ``TECHAPI_DATA_DIR`` environment variable.
 """
 
 from __future__ import annotations
 
 import json
-import os
 import re
 import sys
 from pathlib import Path
 from typing import Any
 
-_DEFAULT_DATA_DIR = Path(__file__).resolve().parent.parent.parent / "TechAPI" / "data"
-DATA_DIR = Path(os.environ.get("TECHAPI_DATA_DIR", _DEFAULT_DATA_DIR))
+from app.data_root import get_data_root
+
+DATA_DIR = get_data_root()
 
 SLUG_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")
 
-BRAND_REQUIRED = {"slug", "name", "country", "categories"}
+BRAND_REQUIRED = {"slug", "name", "country", "categories", "source_urls"}
 BRAND_CATEGORIES = {
     "smartphone-oem",
     "soc-designer",
@@ -84,7 +83,7 @@ def _load(subdir: str) -> list[tuple[str, dict[str, Any]]]:
     if not path.exists():
         return []
     return [
-        (str(f.relative_to(DATA_DIR)), json.loads(f.read_text(encoding="utf-8")))
+        (str(f.relative_to(DATA_DIR)), json.loads(f.read_text(encoding="utf-8-sig")))
         for f in sorted(path.rglob("*.json"))  # recurse into brand subfolders
     ]
 
@@ -133,6 +132,14 @@ def _check_unique_slugs(
             seen[slug] = fname
 
 
+def _check_source_urls(name: str, record: dict[str, Any], errors: list[str]) -> None:
+    urls = record.get("source_urls")
+    if not isinstance(urls, list) or not urls or not all(
+        isinstance(url, str) and url.startswith(("http://", "https://")) for url in urls
+    ):
+        errors.append(f"{name}: source_urls must be a non-empty list of http(s) URL strings")
+
+
 def validate() -> list[str]:
     errors: list[str] = []
 
@@ -156,6 +163,7 @@ def validate() -> list[str]:
 
     for fname, rec in brands:
         _check_required(fname, rec, BRAND_REQUIRED, errors)
+        _check_source_urls(fname, rec, errors)
         _check_slug(fname, rec.get("slug"), errors)
         if "founded_year" in rec:
             _check_range(fname, "founded_year", rec["founded_year"], 1800, 2100, errors)
@@ -188,6 +196,7 @@ def validate() -> list[str]:
 
     for fname, rec in socs:
         _check_required(fname, rec, SOC_REQUIRED, errors)
+        _check_source_urls(fname, rec, errors)
         _check_slug(fname, rec.get("slug"), errors)
         if "release_date" in rec:
             _check_date(fname, rec["release_date"], errors)
@@ -197,6 +206,7 @@ def validate() -> list[str]:
 
     for fname, rec in phones:
         _check_required(fname, rec, PHONE_REQUIRED, errors)
+        _check_source_urls(fname, rec, errors)
         _check_slug(fname, rec.get("slug"), errors)
         if "release_date" in rec:
             _check_date(fname, rec["release_date"], errors)
@@ -212,6 +222,7 @@ def validate() -> list[str]:
 
     for fname, rec in gpus:
         _check_required(fname, rec, GPU_REQUIRED, errors)
+        _check_source_urls(fname, rec, errors)
         _check_slug(fname, rec.get("slug"), errors)
         if "release_date" in rec:
             _check_date(fname, rec["release_date"], errors)
@@ -225,6 +236,7 @@ def validate() -> list[str]:
     valid_segments = {"desktop", "laptop", "hedt", "server"}
     for fname, rec in cpus:
         _check_required(fname, rec, CPU_REQUIRED, errors)
+        _check_source_urls(fname, rec, errors)
         _check_slug(fname, rec.get("slug"), errors)
         if "release_date" in rec:
             _check_date(fname, rec["release_date"], errors)