perf(e2e): parallelize scenario variants within test files#1956
Open
Stephen Belanger (Qard) wants to merge 4 commits intomainfrom
Open
perf(e2e): parallelize scenario variants within test files#1956Stephen Belanger (Qard) wants to merge 4 commits intomainfrom
Stephen Belanger (Qard) wants to merge 4 commits intomainfrom
Conversation
Each multi-variant e2e scenario (e.g. anthropic: 6 SDK versions, ai-sdk: 4 versions) previously ran all describe blocks sequentially inside one Vitest file. Since each withScenarioHarness call starts an isolated mock server on its own ephemeral port with a unique testRunId, the tests are already concurrency-safe. Wrap the outer for-loops in describe.concurrent so all SDK versions run their beforeAll subprocess hooks in parallel (bounded by maxConcurrency: 5). Inner describe.sequential blocks keep "wrapped" and "auto-hook" variants for the same version sequential, avoiding snapshot write races on --update runs. Also parallelize test:external in js/: the isolated ai-sdk-v5, ai-sdk-v6, and claude-agent-sdk suites (each with their own node_modules) now run concurrently via a new cross-platform run-parallel.mjs script, while openai/anthropic/google-genai (shared node_modules + pnpm prune) remain sequential. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nd shell:true in run-parallel - Add describe.sequential around each defineClaudeAgentSDKInstrumentationAssertions call so toMatchFileSnapshot has a test context when describe.concurrent propagates concurrency into the test bodies - Use shell:true in run-parallel.mjs so pnpm.cmd works on Windows Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
for-loop in each multi-variant e2e scenario file indescribe.concurrentso all SDK versions run theirbeforeAllsubprocess hooks in parallel, bounded bymaxConcurrency: 5invitest.config.mtsdescribe.sequentialfor the per-version inner block to keep "wrapped" and "auto-hook" variants sequential within each version, preventing snapshot write races on--updatejs/scripts/run-parallel.mjs, a cross-platform Node.js script that runs the isolated provider test suites (ai-sdk-v5,ai-sdk-v6,claude-agent-sdk) concurrently; the shared-workspace suites (openai, anthropic, google-genai) stay sequential since they callpnpm pruneon a sharednode_modulesExpected CI wins:
anthropic-instrumentation: 6 SDK versions now run in parallel (was sequential)ai-sdk-instrumentation: 4 SDK versions in parallelopenai-instrumentation: 3 versions in parallelclaude-agent-sdk-instrumentation: 4 versions in parallel within each of the two outer groupstest:externalprovider suite: ai-sdk v5/v6 and claude-agent-sdk now overlap with each other instead of chainingAll within the existing CI job structure — no new GitHub Actions jobs.
Test plan
pnpm test:e2e:hermetic— 13 passed, 908 skipped awaiting API keys)js-provider-testsCI run to confirmrun-parallel.mjsexit propagation works on both ubuntu and windows🤖 Generated with Claude Code