ci: add shm transport lane (matrix); document http worker gap

Run the integration suite as a transport matrix:
- launch: the AF_UNIX worker pool (as before).
- shm: launch + the POSIX shared-memory side channel (VGI_RPC_SHM_SIZE_BYTES).
  Validated locally at 171 pass / 13 skip / 0 fail (identical to launch).

run-integration.sh gains a TRANSPORT=launch|http switch and a
VGI_RPC_SHM_SIZE_BYTES passthrough. TRANSPORT=http works locally but is NOT a
CI lane: comparing against the canonical vgi-python http worker showed it passes
sequence.test (table-function streaming) over http while the Java worker errors
on it. The haybarn runner's built-in skip_error_messages HTTP policy masks those
as skips, so an http run looks green while silently dropping ~table tests. The
Java HTTP streaming path needs a fix before http can be a real lane; documented
in ci/README.md and the script.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: rustyconover

.github/workflows/integration.yml

@@ -36,6 +36,15 @@ env:
 jobs:
   integration:
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # launch: the AF_UNIX worker pool. shm: same, plus the POSIX
+          # shared-memory side channel (VGI_RPC_SHM_SIZE_BYTES).
+          - { lane: launch }
+          - { lane: shm, shm_bytes: '67108864' }
+    name: integration (${{ matrix.lane }})
     steps:
       - name: Checkout vgi-java
         uses: actions/checkout@v4
@@ -78,10 +87,12 @@ jobs:
         env:
           GH_TOKEN: ${{ github.token }}
 
-      - name: Run integration suite
+      - name: Run integration suite (${{ matrix.lane }})
         run: ci/run-integration.sh
         env:
           VGI_SRC: ${{ github.workspace }}/vgi-upstream
           VGI_WORKER_BIN: ${{ github.workspace }}/vgi-example-worker/build/install/vgi-example-worker/bin/vgi-example-worker
+          # Empty for the launch lane; run-integration.sh drops an empty value.
+          VGI_RPC_SHM_SIZE_BYTES: ${{ matrix.shm_bytes || '' }}
           # JDK 25 + native access for the worker JVM (FFM shm, haybarn_jdbc).
           JAVA_TOOL_OPTIONS: --enable-native-access=ALL-UNNAMED

ci/README.md

@@ -29,6 +29,25 @@ extension from the Haybarn community channel:
    [`wrappers/`](wrappers), warms the extension cache once, then runs each
    `.test` and tallies pass / skip / fail.
 
+## Transport lanes
+
+The workflow runs a matrix over transports (`run-integration.sh` honours
+`TRANSPORT=launch|http` and a `VGI_RPC_SHM_SIZE_BYTES` passthrough):
+
+- **`launch`** — the AF_UNIX worker pool (default).
+- **`shm`** — `launch` plus the POSIX shared-memory side channel
+  (`VGI_RPC_SHM_SIZE_BYTES`); needs JDK 22+ (`--enable-native-access`).
+
+**`http` is intentionally *not* a CI lane yet.** `TRANSPORT=http` works locally
+(it boots the worker with `--http` and attaches over `http://`), but the Java
+worker's HTTP transport has a known gap: **table-function streaming** (e.g.
+`example.sequence(...)`) errors over http where the vgi-python worker succeeds.
+The haybarn runner has a built-in `skip_error_messages HTTP` policy that turns
+those errors into *skips*, so an http run looks green (0 fail) while silently
+dropping the broken table tests. Promoting http to the matrix should wait until
+that streaming path is fixed. (`subprocess` was skipped as low-value — same
+logic as `launch` but with per-ATTACH JVM cold-start.)
+
 Out of scope and excluded: `writable/`, `simple_writable/` (the port is
 read-only), `nested_type_combinations.test` (segfaults the upstream runner —
 see the project `CLAUDE.md`), and `bool_in_union.test` (characterizes a

ci/run-integration.sh

@@ -39,16 +39,57 @@ mkdir -p "$STAGE/test/sql/integration"
     awk -f "$HERE/preprocess-require.awk" "$f" > "$STAGE/test/sql/integration/$f"
   done )
 
-# launch: amortises the JVM cold-start across the whole run via a flock-coordinated
-# AF_UNIX worker pool. VGI_TEST_DEDICATED_WORKER is a plain (non-pooled) worker for
-# the crash/pool-recovery tests. The three wrappers route the one binary into the
-# versioned / versioned_tables / attach_options catalogs.
+# Transport selection (TRANSPORT=launch|http; default launch):
+#   launch — flock-coordinated AF_UNIX worker pool, amortising JVM cold-start
+#            across the run. Set VGI_RPC_SHM_SIZE_BYTES to also exercise the
+#            shared-memory side channel (the `shm` lane).
+#   http   — boot the worker as an HTTP server and attach over http:// (mirrors
+#            vgi's `make test_http`).
 export VGI_WORKER_BIN
-export VGI_TEST_WORKER="launch:${VGI_WORKER_BIN}"
-export VGI_TEST_DEDICATED_WORKER="${VGI_WORKER_BIN}"
-export VGI_VERSIONED_WORKER="launch:${HERE}/wrappers/vgi-worker-versioned"
-export VGI_VERSIONED_TABLES_WORKER="launch:${HERE}/wrappers/vgi-worker-versioned-tables"
-export VGI_ATTACH_OPTIONS_WORKER="launch:${HERE}/wrappers/vgi-worker-attach-options"
+TRANSPORT="${TRANSPORT:-launch}"
+# An empty VGI_RPC_SHM_SIZE_BYTES must not reach the C++ client (it would try to
+# attach a zero-size segment); only a real value enables the shm side channel.
+[ -n "${VGI_RPC_SHM_SIZE_BYTES:-}" ] || unset VGI_RPC_SHM_SIZE_BYTES
+
+case "$TRANSPORT" in
+  launch)
+    # VGI_TEST_DEDICATED_WORKER is a plain (non-pooled) worker for the crash/
+    # pool-recovery tests; the three wrappers route the one binary into the
+    # versioned / versioned_tables / attach_options catalogs.
+    export VGI_TEST_WORKER="launch:${VGI_WORKER_BIN}"
+    export VGI_TEST_DEDICATED_WORKER="${VGI_WORKER_BIN}"
+    export VGI_VERSIONED_WORKER="launch:${HERE}/wrappers/vgi-worker-versioned"
+    export VGI_VERSIONED_TABLES_WORKER="launch:${HERE}/wrappers/vgi-worker-versioned-tables"
+    export VGI_ATTACH_OPTIONS_WORKER="launch:${HERE}/wrappers/vgi-worker-attach-options"
+    ;;
+  http)
+    # Boot the example worker as an HTTP server on an ephemeral port; it prints
+    # `PORT:<n>` once bound (same readiness contract as vgi-python's
+    # vgi-fixture-http). For LOCAL use — NOT yet a CI lane: the Java worker's
+    # HTTP transport has a known gap (table-function streaming, e.g.
+    # `example.sequence(...)`, errors over http where vgi-python succeeds; the
+    # haybarn runner's built-in `skip_error_messages HTTP` policy masks those as
+    # skips, so the lane looks green while silently dropping ~table tests). Fix
+    # the http streaming path before promoting this to the CI matrix. See
+    # ci/README.md.
+    HTTP_LOG="$(mktemp)"
+    "$VGI_WORKER_BIN" --http --port 0 >"$HTTP_LOG" 2>&1 &
+    HTTP_PID=$!
+    trap '[ -n "${HTTP_PID:-}" ] && kill "$HTTP_PID" 2>/dev/null || true' EXIT
+    PORT=""
+    for _ in $(seq 1 60); do
+      kill -0 "$HTTP_PID" 2>/dev/null || { echo "::error::http worker exited"; cat "$HTTP_LOG"; exit 1; }
+      PORT="$(sed -n 's/.*PORT:\([0-9]*\).*/\1/p' "$HTTP_LOG" | head -1)"
+      [ -n "$PORT" ] && break
+      sleep 0.5
+    done
+    [ -n "$PORT" ] || { echo "::error::http worker never reported a port"; cat "$HTTP_LOG"; exit 1; }
+    echo "HTTP worker on port $PORT (pid $HTTP_PID)"
+    export VGI_TEST_WORKER="http://localhost:${PORT}"
+    ;;
+  *)
+    echo "::error::unknown TRANSPORT=$TRANSPORT (expected launch|http)"; exit 1 ;;
+esac
 
 cd "$STAGE"