Add capability-based cross-engine test selection - #344
Open
vincebln2 wants to merge 12 commits into
Open
Conversation
Collaborator
Author
|
CI failures here are not caused by the capability changes. Wildcard discovery picked up tests/cli/viewport-window.test.js, which was never listed in the Makefile and has never run in CI. Running it exposed a real daemon bug: window resize fails after the active page is closed (see #346 ). That bug will be fixed in a separate PR first, then this branch gets rebased so it stays test-only. |
The client sends the onWebSocket subscribe command without awaiting it, so a quick createWS can beat the subscription and its one-shot created event is lost; no amount of waiting afterwards recovers it. That surfaced as test_fires collecting zero events under full-suite load. Probe with throwaway sockets until one is tracked, which proves the subscription is active, then run the real test flow. Delivery waits poll with a bounded timeout instead of sleeping a fixed interval. The absence check keeps its fixed wait, since polling cannot prove a non-event.
The click is fired without awaiting because the alert blocks it, so a click that fails under load is swallowed and the poll can only time out. Retry the click every 1.5s and extend the poll.
The Firefox CI job never got past the CLI stage before, so the client suites had not run on Firefox. Running them surfaced two engine differences: - Firefox keeps the startup tab in the parent process until its first navigation and refuses script-backed commands there. Navigate it once in the shared fixtures, the tutorial runners, and the Java setups. - Console, error, download, popup, and navigation-capture events are not delivered by the native Firefox path yet. Declare them as chrome-only capabilities and mark the affected tests across the Node, pytest, and JUnit suites. Also wait for websocket OPEN before send in the Python tests: the subscription barrier returns at the created event, which precedes OPEN, and send() on a CONNECTING socket throws.
vincebln2
force-pushed
the
feat/capability-based-test-selection
branch
from
August 10, 2026 23:07
e02150d to
47860ab
Compare
The focused Firefox suite includes Chrome-contrast tests for video recording behavior, so the job needs both browsers. On main those tests ran in the Chrome job, which installs Chrome.
waitForLoad can return before the click's navigation starts, so the URL assertion raced on slower engines. Wait for the subpage URL.
The two tests asserting Chrome's no-video behavior lived in the focused Firefox suite, which now runs in a Firefox-only CI job where Chrome is not installed. Move them to their own file listed under test-js-async so the Chrome job runs them, and drop the Chrome install from the Firefox job again.
Branch protection requires a check named "test", which was the CI job before the chrome/firefox split, so the PR hangs on a check that never reports. The gate depends on both jobs and runs on failure too: a skipped required check would count as satisfied.
The source check accepted any @RequiresCapability in the file, so a class with only method-level markers passed while -PcapabilityOnly, which selects on the class-level tag, silently dropped its unmarked methods. The marker must now sit on the class declaration itself.
The chrome CI job has no Firefox, so the target only produced skips there. The firefox job runs it directly with VIBIUM_REQUIRE_FIREFOX set, which is unchanged. Locally it stays available as make test-firefox.
This was referenced Aug 11, 2026
The download setup command is sent in the background, so a click right after subscribing can start a download before the browser is told where to save it, and the event is lost for good; that failed the chrome CI job under full-suite load. Click in a bounded loop until the download is tracked, the same shape as the websocket fix. The unawaited setup is VibiumDev#351; fixing it lets the tutorial go back to a single click.
After the CI job split, tests/py/test_firefox.py ran nowhere: the chrome job skipped it for lack of Firefox and the firefox job never invoked the full Python sweep. Move it the same way as the test-firefox target: ignored in test-python, run by test-firefox where Firefox is installed and VIBIUM_REQUIRE_FIREFOX turns skips into failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds capability-based test selection so the cross-engine suites run on any engine and skip what the engine cannot do, with a reason. Chrome keeps full coverage. Closes #341.
How it works
tests/capabilities.jsonmaps each capability to the engines that support it.tests/helpers/capabilities.js(Node), capability markers intests/py/conftest.py(pytest),@RequiresCapabilitywith a JUnit condition (Java).-PcapabilityOnlyselects on the class-level tag.make test-capability-audit) fails if Chrome would skip a capability another engine supports, so the full suite cannot silently shrink.CI
test-firefoxleftmake test: the chrome job has no Firefox, so it only produced skips there. The firefox job runs the target directly withVIBIUM_REQUIRE_FIREFOXset so skips fail. The Python Firefox tests (tests/py/test_firefox.py) moved into that target the same way, so they still run somewhere in CI.Firefox findings
docs/how-to-guides/using-firefox.mdmatches.Also in this PR
testClickwaits for the destination URL after a navigating click. The first three work around Event setup commands are sent without awaiting, so a fast trigger loses events #351.firefox.test.jstotests/js/async/chrome-video.test.js, run by the chrome job.Follow-ups: #349, #350, #351.