Skip to content

Commit 08f6e3a

Browse files
Seungpyo1007Seungpyo1007
committed
feat(ingest): weekly crawler scaffolding with Wikipedia CPU source
Closes #2 for v1 scope. Drafts new SKUs from Wikipedia CPU list pages, dedups against the curated dataset, writes additions to a TechAPI checkout, and opens a PR for human review. Modules - app/ingest/normalize.py — frequency/TDP/cache/date/cores parsers + a CPU-segment classifier. - app/ingest/sources/base.py — IngestCandidate dataclass + Protocol. - app/ingest/sources/wikipedia_cpu.py — per-row extraction from the ``List_of_*_processors`` table family with header-keyword column matching and ``<h2>``/``<h3>`` architecture inference. - app/ingest/pipeline.py — runs candidates through curated-set diffing, required-field gating, dedup, and disk write; emits a Markdown summary. - app/ingest/__main__.py — CLI: --category, --limit, --data-root, --summary, --include-drafts, --dry-run. Workflow - .github/workflows/weekly-ingest.yml — Mondays 06:29 UTC (after coverage). Checks out Seungpyo1007/TechAPI under TechAPI/, runs the ingest into data/, and (when TECHAPI_PR_TOKEN is set) creates a branch + PR back to TechAPI. Without the token, attaches a summary artifact and warns. Tests (35 passing) — value parsers, full row extraction with vendored HTML, pipeline behavior (writes, dedups vs curated, skips incomplete unless --include-drafts, dry-run, summary content). Also bumps the engine site landing to mark coverage + ingest as shipped and lists the new workflow.
1 parent 7891bfb commit 08f6e3a

13 files changed

Lines changed: 1047 additions & 12 deletions

File tree

‎.github/workflows/weekly-ingest.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
name: weekly-ingest
2+
3+
# Weekly: scrape upstream catalogs, draft missing SKUs into a TechAPI worktree,
4+
# open a PR for curator review.
5+
on:
6+
schedule:
7+
- cron: "29 6 * * 1" # Mondays 06:29 UTC, after coverage-report (06:23)
8+
workflow_dispatch:
9+
inputs:
10+
category:
11+
description: "Category to ingest"
12+
type: choice
13+
options: [cpu]
14+
default: cpu
15+
limit:
16+
description: "Max candidates per source"
17+
type: string
18+
default: "50"
19+
include_drafts:
20+
description: "Write incomplete records too (PR marked as draft)"
21+
type: boolean
22+
default: false
23+
24+
permissions:
25+
contents: read
26+
27+
jobs:
28+
ingest:
29+
runs-on: ubuntu-latest
30+
steps:
31+
- uses: actions/checkout@v4
32+
33+
# Use the PAT when present so we can push to the TechAPI fork later;
34+
# fall back to the default token for read-only test runs.
35+
- uses: actions/checkout@v4
36+
with:
37+
repository: Seungpyo1007/TechAPI
38+
path: TechAPI
39+
token: ${{ secrets.TECHAPI_PR_TOKEN || secrets.GITHUB_TOKEN }}
40+
41+
- uses: actions/setup-python@v5
42+
with:
43+
python-version: "3.12"
44+
cache: pip
45+
46+
- name: Install
47+
run: pip install -e .
48+
49+
- name: Run ingest
50+
env:
51+
TECHAPI_DATA_DIR: ${{ github.workspace }}/TechAPI/data
52+
INGEST_CATEGORY: ${{ inputs.category || 'cpu' }}
53+
INGEST_LIMIT: ${{ inputs.limit || '50' }}
54+
INGEST_DRAFTS: ${{ inputs.include_drafts && '--include-drafts' || '' }}
55+
run: |
56+
python -m app.ingest \
57+
--category "$INGEST_CATEGORY" \
58+
--limit "$INGEST_LIMIT" \
59+
--data-root TechAPI/data \
60+
--summary ingest-summary.md \
61+
$INGEST_DRAFTS
62+
63+
- name: Upload summary artifact
64+
uses: actions/upload-artifact@v4
65+
with:
66+
name: ingest-summary
67+
path: ingest-summary.md
68+
69+
- name: Check whether ingest produced any additions
70+
id: changes
71+
run: |
72+
cd TechAPI
73+
if [ -n "$(git status --porcelain)" ]; then
74+
echo "has_changes=true" >> "$GITHUB_OUTPUT"
75+
else
76+
echo "has_changes=false" >> "$GITHUB_OUTPUT"
77+
fi
78+
79+
- name: Open PR against TechAPI
80+
if: ${{ steps.changes.outputs.has_changes == 'true' && secrets.TECHAPI_PR_TOKEN != '' }}
81+
env:
82+
GH_TOKEN: ${{ secrets.TECHAPI_PR_TOKEN }}
83+
CATEGORY: ${{ inputs.category || 'cpu' }}
84+
IS_DRAFT: ${{ inputs.include_drafts && 'true' || 'false' }}
85+
run: |
86+
set -euo pipefail
87+
cd TechAPI
88+
BRANCH="ingest/${CATEGORY}-$(date -u +%Y%m%d-%H%M%S)"
89+
git config user.name "techengine-bot"
90+
git config user.email "techengine-bot@users.noreply.github.com"
91+
git checkout -b "$BRANCH"
92+
git add data/
93+
git commit -m "feat(data/${CATEGORY}): weekly ingest"
94+
git push -u origin "$BRANCH"
95+
DRAFT_FLAG=""
96+
if [ "$IS_DRAFT" = "true" ]; then
97+
DRAFT_FLAG="--draft"
98+
fi
99+
gh pr create \
100+
--title "feat(data/${CATEGORY}): weekly ingest" \
101+
--body-file ../ingest-summary.md \
102+
--base main \
103+
--head "$BRANCH" \
104+
$DRAFT_FLAG
105+
106+
- name: Note when PR token is missing
107+
if: ${{ steps.changes.outputs.has_changes == 'true' && secrets.TECHAPI_PR_TOKEN == '' }}
108+
run: |
109+
echo "::warning::Ingest produced additions but TECHAPI_PR_TOKEN is unset; skipping PR. Summary attached as artifact."

‎README.md‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,13 +25,17 @@ app/
2525
├ models/ # SQLModel tables
2626
├ routers/ # /v1/{brands,socs,smartphones,gpus,cpus}
2727
├ schemas/ # Pydantic response models
28-
└ services/ # scoring (algorithm_version-tagged)
28+
├ services/ # scoring (algorithm_version-tagged)
29+
├ coverage/ # upstream-vs-curated diff + Markdown report
30+
└ ingest/ # draft new records from upstream pages
2931
tests/ # unit + integration
3032
site/ # Astro engine landing (deploys to Pages)
3133
docs/ # SPEC / DATA_PIPELINE / DEVELOPMENT
3234
.github/workflows/
3335
├ validate-data.yml # workflow_call: PR-time data validation for TechAPI
3436
├ refresh-data.yml # cron: regenerate the static dump weekly
37+
├ coverage-report.yml # cron: gap report, sticky issue
38+
├ weekly-ingest.yml # cron: drafts new SKUs, opens PR against TechAPI
3539
├ deploy-pages.yml # build & deploy engine site + dump
3640
└ test.yml # lint + type-check + tests
3741
```
@@ -74,11 +78,13 @@ Spins up Postgres 16, seeds from the mounted TechAPI checkout, serves on `:8000`
7478
## Roadmap
7579

7680
- [x] Split out from TechAPI; sibling-checkout data pipeline
77-
- [ ] **Coverage gap detector** — diff curated dataset vs upstream catalogs
78-
(Intel ARK, AMD product pages, Wikipedia infoboxes, TechPowerUp DB) and emit
79-
weekly issues listing missing SKUs ([#1](https://github.com/GetTechAPI/TechEngine/issues/1))
80-
- [ ] **Weekly ingestion crawler** — scrape canonical sources and open PRs
81-
against TechAPI with new SKUs ([#2](https://github.com/GetTechAPI/TechEngine/issues/2))
81+
- [x] **Coverage gap detector** — diff curated dataset vs upstream catalogs
82+
and surface missing SKUs as a sticky weekly issue
83+
([#1](https://github.com/GetTechAPI/TechEngine/issues/1))
84+
- [x] **Weekly ingestion crawler** — scrape canonical sources and open PRs
85+
against TechAPI with new SKUs (requires `TECHAPI_PR_TOKEN` secret to push)
86+
([#2](https://github.com/GetTechAPI/TechEngine/issues/2))
87+
- [ ] More sources (Intel ARK, AMD product pages, TechPowerUp DB)
8288

8389
## License
8490

‎app/ingest/__init__.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
"""Automated ingestion crawler.
2+
3+
Reads upstream catalogs (Wikipedia list pages today, vendor product pages
4+
later), normalizes each row into a TechAPI-shaped JSON record, and writes
5+
draft records into a TechAPI checkout. A companion CI workflow opens a PR
6+
against TechAPI so curators only review.
7+
8+
Entry point: ``python -m app.ingest --category cpu --limit 5``.
9+
"""

‎app/ingest/__main__.py‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
"""Ingestion CLI.
2+
3+
::
4+
5+
python -m app.ingest --category cpu --limit 5 \\
6+
--data-root ../TechAPI/data --summary ingest-summary.md
7+
8+
Streams candidates from every wired source for ``--category``, dedups
9+
against the curated dataset, writes additions to ``--data-root``, and
10+
emits a Markdown summary (used by the weekly workflow as the PR body).
11+
"""
12+
13+
from __future__ import annotations
14+
15+
import argparse
16+
import os
17+
import sys
18+
from collections.abc import Iterator
19+
from pathlib import Path
20+
21+
from .pipeline import run
22+
from .sources.base import IngestCandidate, IngestSource
23+
from .sources.wikipedia_cpu import WikipediaCpuIngest
24+
25+
SOURCES_BY_CATEGORY: dict[str, list[IngestSource]] = {
26+
"cpu": [WikipediaCpuIngest()],
27+
}
28+
29+
30+
def _default_data_root() -> Path:
31+
explicit = os.environ.get("TECHAPI_DATA_DIR")
32+
if explicit:
33+
return Path(explicit)
34+
return Path(__file__).resolve().parent.parent.parent.parent / "TechAPI" / "data"
35+
36+
37+
def _collect(category: str, limit: int | None) -> Iterator[IngestCandidate]:
38+
for source in SOURCES_BY_CATEGORY.get(category, []):
39+
yield from source.fetch(limit=limit)
40+
41+
42+
def main(argv: list[str] | None = None) -> int:
43+
parser = argparse.ArgumentParser(prog="app.ingest")
44+
parser.add_argument(
45+
"--category", required=True, choices=sorted(SOURCES_BY_CATEGORY.keys())
46+
)
47+
parser.add_argument(
48+
"--limit", type=int, default=None, help="Max candidates to consider per source."
49+
)
50+
parser.add_argument(
51+
"--data-root",
52+
type=Path,
53+
default=_default_data_root(),
54+
help="Path to the TechAPI ``data/`` directory.",
55+
)
56+
parser.add_argument(
57+
"--summary",
58+
type=Path,
59+
default=Path("ingest-summary.md"),
60+
help="Markdown summary destination (PR body).",
61+
)
62+
parser.add_argument(
63+
"--include-drafts",
64+
action="store_true",
65+
help="Write records that are missing required fields too (PR will be a draft).",
66+
)
67+
parser.add_argument(
68+
"--dry-run", action="store_true", help="Compute everything; write nothing."
69+
)
70+
args = parser.parse_args(argv)
71+
72+
candidates = list(_collect(args.category, args.limit))
73+
result = run(
74+
candidates,
75+
data_root=args.data_root,
76+
include_drafts=args.include_drafts,
77+
dry_run=args.dry_run,
78+
)
79+
args.summary.write_text(result.markdown_summary(), encoding="utf-8")
80+
print(
81+
f"category={args.category} considered={len(candidates)} "
82+
f"written={len(result.written)} "
83+
f"skipped_existing={len(result.skipped_existing)} "
84+
f"skipped_incomplete={len(result.skipped_incomplete)}"
85+
)
86+
return 0
87+
88+
89+
if __name__ == "__main__":
90+
sys.exit(main(sys.argv[1:]))

0 commit comments

Comments
 (0)