ci: add http + unix VGI transport coverage to SQL E2E#1
Merged
Conversation
Parameterize ci/run-integration.sh by a TRANSPORT env (subprocess | http | unix, default subprocess) and run the full test/sql/*.test suite over all three VGI transports in a transport×os CI matrix. - The mock MediaWiki server now stays up for ALL transports, and VGI_WIKIPEDIA_API_URL is EXPORTED into the environment (not baked in as an `env VAR=… <cmd>` command prefix). The prefix only worked when DuckDB spawned the worker (subprocess); exporting lets the out-of-band http/unix workers inherit it too. - http: inject `INSTALL httpfs FROM core; LOAD httpfs;` into staged tests (the vgi HTTP transport rides DuckDB's httpfs), boot the worker with `--http --port 0 --port-file <f>`, poll the port-file, ATTACH `http://127.0.0.1:<port>`. - unix: boot the worker with `--unix <sock>`, poll for the socket, ATTACH `unix://<sock>`. - Silent-skip guard: the sqllogictest runner auto-SKIPs HTTP errors (fake green); fail the leg unless it reports `All tests passed (N>0 assertions)` with zero skips. - One cleanup trap captures $? first and kills both the mock server and the worker. Python http extra: add an `http = ["vgi-python[http]>=0.8.3"]` optional dependency, list `vgi-python[http]` in the worker's PEP 723 header, relock, and `uv sync --frozen --extra http` in the integration job. `wiki_search`'s sroffset paging already externalizes its cursor into a serializable ScanState, so it round-trips correctly over the stateless HTTP transport — full suite incl. wiki_search.test runs over http with no gate and no worker change. Local: subprocess/unix 51 assertions, http 55 (51 + injected httpfs) — all GREEN. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds subprocess + http + unix VGI transport coverage to the SQL E2E suite, with the full
test/sql/*.testsuite GREEN over all three transports.What changed
ci/run-integration.shis now parameterized byTRANSPORT(subprocess|http|unix, defaultsubprocess). The mock MediaWiki server stays up for all transports, andVGI_WIKIPEDIA_API_URLis exported into the environment (not baked in as anenv VAR=… <cmd>prefix) so both the DuckDB-spawned subprocess worker and the out-of-band http/unix workers inherit it.INSTALL httpfs FROM core; LOAD httpfs;into staged tests (the vgi HTTP transport rides DuckDB's httpfs), bootswikipedia_worker.py --http --port 0 --port-file <f>, polls the port-file, ATTACHeshttp://127.0.0.1:<port>.wikipedia_worker.py --unix <sock>, polls for the socket, ATTACHesunix://<sock>.All tests passed (N>0 assertions)with zero skips.$?first and kills both the mock server and the worker..github/workflows/ci.yml): theintegrationjob is now atransport: [subprocess, http, unix]×os: [ubuntu-latest, macos-latest]matrix. Test + lint jobs unchanged.http = ["vgi-python[http]>=0.8.3"], listedvgi-python[http]in the worker's PEP 723 header, relocked, anduv sync --frozen --extra httpin the integration job.Streaming over HTTP
wiki_search'ssroffsetpaging already externalizes its cursor into a serializableScanState, so it round-trips correctly over the stateless HTTP transport — the full suite (incl.wiki_search.test'smax_pages := 2batch/page-boundary test) runs over http with no gate and no worker change.Local validation
🤖 Generated with Claude Code