11# TechEngine
22
3- > ** Validation, ingestion, and serving engine for the [ TechAPI] ( https://github.com/Seungpyo1007 /TechAPI ) dataset.**
3+ > ** Validation, ingestion, and serving engine for the [ TechAPI] ( https://github.com/GetTechAPI /TechAPI ) dataset.**
44
55[ ![ test] ( https://github.com/GetTechAPI/TechEngine/actions/workflows/test.yml/badge.svg )] ( https://github.com/GetTechAPI/TechEngine/actions/workflows/test.yml )
6-   ; Code: ** MIT** · Data: lives in ** [ TechAPI] ( https://github.com/Seungpyo1007 /TechAPI ) ** (CC-BY-SA 4.0)
6+   ; Code: ** MIT** · Data: lives in ** [ TechAPI] ( https://github.com/GetTechAPI /TechAPI ) ** (CC-BY-SA 4.0)
77
88TechEngine owns everything * around* the data: schema validation, the FastAPI
99read API, the static JSON dump generator, the engine's own landing site, and
1010(next up) automated coverage checks and a weekly ingestion crawler.
1111
1212The dataset and the public-facing playground site live in
13- [ TechAPI] ( https://github.com/Seungpyo1007 /TechAPI ) so each can be versioned,
13+ [ TechAPI] ( https://github.com/GetTechAPI /TechAPI ) so each can be versioned,
1414mirrored, and licensed independently. The site shipped in this repo is the
1515engine's own landing — what TechEngine is, what it runs, link out to docs.
1616
3131tests/ # unit + integration
3232site/ # Astro engine landing (deploys to Pages)
3333docs/ # SPEC / DATA_PIPELINE / DEVELOPMENT
34+ TechAPI/ # submodule → GetTechAPI/TechAPI (clickable @ <sha> link)
3435.github/workflows/
35- ├ validate-data.yml # workflow_call: PR-time data validation for TechAPI
36- ├ refresh-data.yml # cron: regenerate the static dump weekly
37- ├ coverage-report.yml # cron: gap report, sticky issue
38- ├ weekly-ingest.yml # cron: drafts new SKUs, opens PR against TechAPI
39- ├ deploy-pages.yml # build & deploy engine site + dump
40- └ test.yml # lint + type-check + tests
36+ ├ validate-data.yml # workflow_call: PR-time data validation for TechAPI
37+ ├ weekly-refresh.yml # cron: live-scrape → integrity gate → dump → PR to TechAPI
38+ ├ weekly-ingest.yml # cron: draft new SKUs, open PR against TechAPI
39+ ├ coverage-report.yml # cron: gap report, sticky issue (TechEngine + TechAPI)
40+ ├ refresh-data.yml # smoke-test: rebuild the dump on engine (app/**) changes
41+ ├ notify-techapi.yml # push→main: ping TechAPI to bump its TechEngine submodule
42+ ├ bump-techapi.yml # dispatch: advance this repo's TechAPI submodule pointer
43+ ├ deploy-pages.yml # build & deploy engine site + dump
44+ └ test.yml # lint + type-check + tests
4145```
4246
4347## How the two repos connect
4448
45- ```
46- ┌────────────────────┐ ┌──────────────────────────┐
47- │ TechAPI (data/) │ workflow_call │ TechEngine (this repo) │
48- │ + bundled self- │ ─────────────▶ │ validate-data.yml │
49- │ check (PR) │ ◀───────────── │ (checks out TechAPI) │
50- └────────────────────┘ └──────────────────────────┘
51- ```
49+ Both repos live in the ** GetTechAPI** org and each includes the other as a git
50+ ** submodule** (a clickable ` @ <sha> ` pin). Three automations keep them in step:
51+
52+ - ** validate-data.yml** (` workflow_call ` ) — TechAPI's PR-time check calls into
53+ TechEngine to validate its data.
54+ - ** weekly-refresh.yml** — live-scrapes benchmarks, runs the full-dataset
55+ integrity gate (` app.validate ` + ` integrity_check.py --strict ` ), regenerates
56+ the static dump, and opens a dated refresh PR against TechAPI.
57+ - ** Submodule autosync** — every push to TechEngine ` main ` fires
58+ ` notify-techapi.yml ` , which pings TechAPI to bump its TechEngine pointer;
59+ conversely ` bump-techapi.yml ` advances TechEngine's TechAPI pointer when
60+ TechAPI changes. Bumps are loop-guarded, so each real change converges to one.
5261
53- Every Python entry point reads data from a sibling ** TechAPI checkout** . The
54- location can be overridden via ` TECHAPI_DATA_DIR ` ; the default looks for
55- ` ../TechAPI/data ` next to this repo, which matches a local dev layout.
62+ Every Python entry point reads data from a ** TechAPI checkout** . The location
63+ can be overridden via ` TECHAPI_DATA_DIR ` ; the default looks for ` ../TechAPI/data `
64+ next to this repo, which matches a local dev layout.
5665
5766## Quickstart
5867
5968``` bash
60- git clone https://github.com/Seungpyo1007 /TechAPI.git ../TechAPI # data source
69+ git clone https://github.com/GetTechAPI /TechAPI.git ../TechAPI # data source
6170pip install -e " .[dev]"
6271python -m app.validate # check data integrity
6372python -m app.seed # data/ → ./techapi.db (SQLite)
@@ -82,8 +91,11 @@ Spins up Postgres 16, seeds from the mounted TechAPI checkout, serves on `:8000`
8291 and surface missing SKUs as a sticky weekly issue
8392 ([ #1 ] ( https://github.com/GetTechAPI/TechEngine/issues/1 ) )
8493- [x] ** Weekly ingestion crawler** — scrape canonical sources and open PRs
85- against TechAPI with new SKUs (requires ` TECHAPI_PR_TOKEN ` secret to push)
94+ against TechAPI with new SKUs (requires the ` TECHAPI_TOKEN ` secret to push)
8695 ([ #2 ] ( https://github.com/GetTechAPI/TechEngine/issues/2 ) )
96+ - [x] ** Weekly refresh pipeline** — live benchmark enrichment → full-dataset
97+ integrity gate → static dump → dated refresh PR (` weekly-refresh.yml ` )
98+ - [x] ** Bidirectional submodule autosync** between TechEngine and TechAPI
8799- [ ] More sources (Intel ARK, AMD product pages, TechPowerUp DB)
88100
89101## License
0 commit comments