Skip to content

ci: add http + unix VGI transport coverage to SQL E2E#1

Merged
rustyconover merged 1 commit into
mainfrom
ci/multi-transport-sql
Jun 24, 2026
Merged

ci: add http + unix VGI transport coverage to SQL E2E#1
rustyconover merged 1 commit into
mainfrom
ci/multi-transport-sql

Conversation

@rustyconover

Copy link
Copy Markdown
Contributor

Adds subprocess + http + unix VGI transport coverage to the SQL E2E suite, with the full test/sql/*.test suite GREEN over all three transports.

What changed

  • ci/run-integration.sh is now parameterized by TRANSPORT (subprocess | http | unix, default subprocess). The mock MediaWiki server stays up for all transports, and VGI_WIKIPEDIA_API_URL is exported into the environment (not baked in as an env VAR=… <cmd> prefix) so both the DuckDB-spawned subprocess worker and the out-of-band http/unix workers inherit it.
    • http: injects INSTALL httpfs FROM core; LOAD httpfs; into staged tests (the vgi HTTP transport rides DuckDB's httpfs), boots wikipedia_worker.py --http --port 0 --port-file <f>, polls the port-file, ATTACHes http://127.0.0.1:<port>.
    • unix: boots wikipedia_worker.py --unix <sock>, polls for the socket, ATTACHes unix://<sock>.
    • Silent-skip guard: the sqllogictest runner auto-SKIPs HTTP errors (fake green); the leg fails unless the runner reports All tests passed (N>0 assertions) with zero skips.
    • One cleanup trap captures $? first and kills both the mock server and the worker.
  • CI (.github/workflows/ci.yml): the integration job is now a transport: [subprocess, http, unix] × os: [ubuntu-latest, macos-latest] matrix. Test + lint jobs unchanged.
  • Python http extra: added http = ["vgi-python[http]>=0.8.3"], listed vgi-python[http] in the worker's PEP 723 header, relocked, and uv sync --frozen --extra http in the integration job.

Streaming over HTTP

wiki_search's sroffset paging already externalizes its cursor into a serializable ScanState, so it round-trips correctly over the stateless HTTP transport — the full suite (incl. wiki_search.test's max_pages := 2 batch/page-boundary test) runs over http with no gate and no worker change.

Local validation

  • subprocess: GREEN — 51 assertions
  • unix: GREEN — 51 assertions
  • http: GREEN — 55 assertions (51 + injected httpfs INSTALL/LOAD)

🤖 Generated with Claude Code

Parameterize ci/run-integration.sh by a TRANSPORT env (subprocess | http |
unix, default subprocess) and run the full test/sql/*.test suite over all
three VGI transports in a transport×os CI matrix.

- The mock MediaWiki server now stays up for ALL transports, and
  VGI_WIKIPEDIA_API_URL is EXPORTED into the environment (not baked in as an
  `env VAR=… <cmd>` command prefix). The prefix only worked when DuckDB
  spawned the worker (subprocess); exporting lets the out-of-band http/unix
  workers inherit it too.
- http: inject `INSTALL httpfs FROM core; LOAD httpfs;` into staged tests
  (the vgi HTTP transport rides DuckDB's httpfs), boot the worker with
  `--http --port 0 --port-file <f>`, poll the port-file, ATTACH
  `http://127.0.0.1:<port>`.
- unix: boot the worker with `--unix <sock>`, poll for the socket, ATTACH
  `unix://<sock>`.
- Silent-skip guard: the sqllogictest runner auto-SKIPs HTTP errors (fake
  green); fail the leg unless it reports `All tests passed (N>0 assertions)`
  with zero skips.
- One cleanup trap captures $? first and kills both the mock server and the
  worker.

Python http extra: add an `http = ["vgi-python[http]>=0.8.3"]` optional
dependency, list `vgi-python[http]` in the worker's PEP 723 header, relock,
and `uv sync --frozen --extra http` in the integration job.

`wiki_search`'s sroffset paging already externalizes its cursor into a
serializable ScanState, so it round-trips correctly over the stateless HTTP
transport — full suite incl. wiki_search.test runs over http with no gate and
no worker change.

Local: subprocess/unix 51 assertions, http 55 (51 + injected httpfs) — all GREEN.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rustyconover rustyconover merged commit 469f6de into main Jun 24, 2026
10 checks passed
@rustyconover rustyconover deleted the ci/multi-transport-sql branch June 24, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant