Skip to content

Commit 5615376

Browse files
committed
docs: reflect org move, weekly-refresh pipeline, and submodule autosync
Update README + DATA_PIPELINE + DEVELOPMENT for the current state: TechAPI now at GetTechAPI/TechAPI, the weekly-refresh pipeline (enrich → integrity gate → dump → PR) as the real weekly automation with refresh-data demoted to a dump smoke-test, in-repo collection (ingest/enrich), the bidirectional submodule autosync (notify-techapi / bump-techapi), the new workflow list, and the TECHAPI_PR_TOKEN → TECHAPI_TOKEN rename.
1 parent f5ca03e commit 5615376

3 files changed

Lines changed: 55 additions & 32 deletions

File tree

README.md

Lines changed: 33 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,16 @@
11
# TechEngine
22

3-
> **Validation, ingestion, and serving engine for the [TechAPI](https://github.com/Seungpyo1007/TechAPI) dataset.**
3+
> **Validation, ingestion, and serving engine for the [TechAPI](https://github.com/GetTechAPI/TechAPI) dataset.**
44
55
[![test](https://github.com/GetTechAPI/TechEngine/actions/workflows/test.yml/badge.svg)](https://github.com/GetTechAPI/TechEngine/actions/workflows/test.yml)
6-
 Code: **MIT** · Data: lives in **[TechAPI](https://github.com/Seungpyo1007/TechAPI)** (CC-BY-SA 4.0)
6+
 Code: **MIT** · Data: lives in **[TechAPI](https://github.com/GetTechAPI/TechAPI)** (CC-BY-SA 4.0)
77

88
TechEngine owns everything *around* the data: schema validation, the FastAPI
99
read API, the static JSON dump generator, the engine's own landing site, and
1010
(next up) automated coverage checks and a weekly ingestion crawler.
1111

1212
The dataset and the public-facing playground site live in
13-
[TechAPI](https://github.com/Seungpyo1007/TechAPI) so each can be versioned,
13+
[TechAPI](https://github.com/GetTechAPI/TechAPI) so each can be versioned,
1414
mirrored, and licensed independently. The site shipped in this repo is the
1515
engine's own landing — what TechEngine is, what it runs, link out to docs.
1616

@@ -31,33 +31,42 @@ app/
3131
tests/ # unit + integration
3232
site/ # Astro engine landing (deploys to Pages)
3333
docs/ # SPEC / DATA_PIPELINE / DEVELOPMENT
34+
TechAPI/ # submodule → GetTechAPI/TechAPI (clickable @ <sha> link)
3435
.github/workflows/
35-
├ validate-data.yml # workflow_call: PR-time data validation for TechAPI
36-
├ refresh-data.yml # cron: regenerate the static dump weekly
37-
├ coverage-report.yml # cron: gap report, sticky issue
38-
├ weekly-ingest.yml # cron: drafts new SKUs, opens PR against TechAPI
39-
├ deploy-pages.yml # build & deploy engine site + dump
40-
└ test.yml # lint + type-check + tests
36+
├ validate-data.yml # workflow_call: PR-time data validation for TechAPI
37+
├ weekly-refresh.yml # cron: live-scrape → integrity gate → dump → PR to TechAPI
38+
├ weekly-ingest.yml # cron: draft new SKUs, open PR against TechAPI
39+
├ coverage-report.yml # cron: gap report, sticky issue (TechEngine + TechAPI)
40+
├ refresh-data.yml # smoke-test: rebuild the dump on engine (app/**) changes
41+
├ notify-techapi.yml # push→main: ping TechAPI to bump its TechEngine submodule
42+
├ bump-techapi.yml # dispatch: advance this repo's TechAPI submodule pointer
43+
├ deploy-pages.yml # build & deploy engine site + dump
44+
└ test.yml # lint + type-check + tests
4145
```
4246

4347
## How the two repos connect
4448

45-
```
46-
┌────────────────────┐ ┌──────────────────────────┐
47-
│ TechAPI (data/) │ workflow_call │ TechEngine (this repo) │
48-
│ + bundled self- │ ─────────────▶ │ validate-data.yml │
49-
│ check (PR) │ ◀───────────── │ (checks out TechAPI) │
50-
└────────────────────┘ └──────────────────────────┘
51-
```
49+
Both repos live in the **GetTechAPI** org and each includes the other as a git
50+
**submodule** (a clickable `@ <sha>` pin). Three automations keep them in step:
51+
52+
- **validate-data.yml** (`workflow_call`) — TechAPI's PR-time check calls into
53+
TechEngine to validate its data.
54+
- **weekly-refresh.yml** — live-scrapes benchmarks, runs the full-dataset
55+
integrity gate (`app.validate` + `integrity_check.py --strict`), regenerates
56+
the static dump, and opens a dated refresh PR against TechAPI.
57+
- **Submodule autosync** — every push to TechEngine `main` fires
58+
`notify-techapi.yml`, which pings TechAPI to bump its TechEngine pointer;
59+
conversely `bump-techapi.yml` advances TechEngine's TechAPI pointer when
60+
TechAPI changes. Bumps are loop-guarded, so each real change converges to one.
5261

53-
Every Python entry point reads data from a sibling **TechAPI checkout**. The
54-
location can be overridden via `TECHAPI_DATA_DIR`; the default looks for
55-
`../TechAPI/data` next to this repo, which matches a local dev layout.
62+
Every Python entry point reads data from a **TechAPI checkout**. The location
63+
can be overridden via `TECHAPI_DATA_DIR`; the default looks for `../TechAPI/data`
64+
next to this repo, which matches a local dev layout.
5665

5766
## Quickstart
5867

5968
```bash
60-
git clone https://github.com/Seungpyo1007/TechAPI.git ../TechAPI # data source
69+
git clone https://github.com/GetTechAPI/TechAPI.git ../TechAPI # data source
6170
pip install -e ".[dev]"
6271
python -m app.validate # check data integrity
6372
python -m app.seed # data/ → ./techapi.db (SQLite)
@@ -82,8 +91,11 @@ Spins up Postgres 16, seeds from the mounted TechAPI checkout, serves on `:8000`
8291
and surface missing SKUs as a sticky weekly issue
8392
([#1](https://github.com/GetTechAPI/TechEngine/issues/1))
8493
- [x] **Weekly ingestion crawler** — scrape canonical sources and open PRs
85-
against TechAPI with new SKUs (requires `TECHAPI_PR_TOKEN` secret to push)
94+
against TechAPI with new SKUs (requires the `TECHAPI_TOKEN` secret to push)
8695
([#2](https://github.com/GetTechAPI/TechEngine/issues/2))
96+
- [x] **Weekly refresh pipeline** — live benchmark enrichment → full-dataset
97+
integrity gate → static dump → dated refresh PR (`weekly-refresh.yml`)
98+
- [x] **Bidirectional submodule autosync** between TechEngine and TechAPI
8799
- [ ] More sources (Intel ARK, AMD product pages, TechPowerUp DB)
88100

89101
## License

docs/DATA_PIPELINE.md

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -44,23 +44,32 @@ dump/v1/socs/… /v1/gpus/… /v1/cpus/… /v1/brands/…
4444
A static consumer just fetches, e.g.
4545
`https://<host>/v1/smartphones/galaxy-s25/index.json`.
4646

47-
## 3. Automated refresh (`.github/workflows/refresh-data.yml`)
47+
## 3. Automated refresh (`.github/workflows/weekly-refresh.yml`)
4848

49-
A scheduled workflow (weekly cron + on `data/**` changes + manual) runs:
49+
The weekly pipeline (Monday cron + manual dispatch) runs the full cycle against
50+
a TechAPI checkout:
5051

5152
```
52-
validate seed data → generate dump → publish/commit dump if changed
53+
live-scrape benchmark sources → full-dataset integrity gate
54+
(app.validate + integrity_check.py --strict) → regenerate static dump
55+
→ open a dated refresh PR against TechAPI
5356
```
5457

55-
This is the git-scraping pattern: GitHub runs and stores everything for free.
56-
The hosting target depends on the public/private decision (§5).
58+
The integrity gate re-checks the **whole** dataset every run (not just new
59+
rows), so a bad scrape can't slip a contaminated value past it. A lighter
60+
`refresh-data.yml` rebuilds the dump on engine (`app/**`) changes as a smoke
61+
test only. This is the git-scraping pattern — GitHub runs and stores everything
62+
for free — and the dated PR keeps every refresh reviewable before it lands. The
63+
hosting target depends on the public/private decision (§5).
5764

5865
## 4. Where the data comes from
5966

60-
This repo contains only **curated, validated** records. Bulk collection and
61-
normalization happen **outside this repo**, through a separate internal pipeline,
62-
which publishes curated records here (by PR) after review (SPEC §9.3). This repo
63-
never needs scraping/browser dependencies.
67+
This repo serves **curated, validated** records, but collection now happens
68+
**in-repo**: `app/ingest` drafts new SKUs from upstream catalogs and
69+
`app/ingest/enrich` backfills benchmark columns from multiple sources
70+
(variant-safe, fill-only-nulls, never overwrites). Both run weekly and open PRs
71+
against TechAPI for human review before anything lands (SPEC §9.3). The curated
72+
dataset is a **subset, not exhaustive.**
6473

6574
**Dataset layout (this repo).** Curated data uses singular folder names and is
6675
organised by brand: `data/brand/<slug>.json`, `data/soc/<manufacturer>/<slug>.json`,

docs/DEVELOPMENT.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,9 @@ scripts/
6060
dump.py DB → static JSON dump (replays API in-process) → ./dump
6161
tests/ unit/ + integration/ (conftest seeds a temp SQLite from data/)
6262
docs/ SPEC.md, DATA_PIPELINE.md, DEVELOPMENT.md
63-
.github/workflows/ test.yml, validate-data.yml, refresh-data.yml
63+
.github/workflows/ test.yml, validate-data.yml, weekly-refresh.yml, weekly-ingest.yml,
64+
coverage-report.yml, refresh-data.yml, notify-techapi.yml,
65+
bump-techapi.yml, deploy-pages.yml
6466
```
6567

6668
> Note: **data folders are singular** (`data/soc/…`) but **API routes are plural**
@@ -85,7 +87,7 @@ python -m app.dump # generate ./dump/ static tree (gitignored)
8587
- **GPU activated** — model existed (§6.5); endpoints + data added.
8688
- **Data restructured** to singular names + brand subfolders (maintainer request).
8789
- **Static-dump pivot**`app/dump.py` exports the API to a static JSON tree,
88-
refreshed by GitHub Actions (`refresh-data.yml`).
90+
refreshed weekly by GitHub Actions (`weekly-refresh.yml`).
8991
- **Scoring** is a Phase-0 reference-based approximation; Phase 1 → dataset-wide
9092
min-max (§8.4). Raw third-party benchmarks (Geekbench/AnTuTu/Cinebench/Time Spy)
9193
are stored as algorithm inputs but NOT exposed (ADR-006).

0 commit comments

Comments
 (0)