[WIP] Another attempt on adding a headless runner

[DmitrySharabin]

Based on Playwright.

Some docs: https://deploy-preview-120--h-test.netlify.app/docs/run/headless/

---

For those who are interested in testing the implementation of the hTest headless runner

Please follow these steps to run your tests in hTest format in the hTest headless runner.

Setup

0. In the terminal, cd path/to/your/project.
1. npm install -D github:htest-dev/htest#headless-runner (to install hTest directly from the branch associated with this PR)
2. npm install -D playwright (we don't provide Playwright as any dependency, so this step is mandatory)
3. Install browser engines (at least one) you want your tests to run in. For example, npx playwright install firefox. Playwright supports the following browser engines: chromium, firefox, webkit, chrome, edge.

Note

If you try to run your tests in a browser that wasn't downloaded later, you'll get an error with the exact command you need to run to install it.

Running tests

Suppose all your tests are in the tests folder inside the project root, and all those tests are supposed to run in the hTest headless runner. In the terminal, run npx htest --headless tests (you can even use this command as an NPM script inside package.json).

Note

When running tests in the headless runner for the first time, it might take some time to spin the browser off. All the next runs will be faster.

To run tests from a particular file, tweak the path in the previous command. For example, npx htest --headless tests/DOMRect.js.

As the default runner, hTest uses Chromium (as Playwright does). If you need your tests to run in another (already downloaded) browser, pass its name (or the name of the engine) as a value of the --browser option. For example, npx htest --headless --browser webkit tests.

Other flags, like --verbose and --ci, are also supported. For example, running npx htest --headless --verbose --ci tests/DOMRect.js, you might get something like this:

Warning

Don't run tests that import NPM packages in the headless runner; it won't work. They will throw during import because of unsupported specifiers (unless you are using something like Nudeps, which will fix it for you).
All other types of tests should correctly run in the headless runner. If not, please let us know what doesn't work.

[netlify]

✅ Deploy Preview for h-test ready!

Name	Link
🔨 Latest commit	bd53fea
🔍 Latest deploy log	https://app.netlify.com/projects/h-test/deploys/697237a54076bc00085257b8
😎 Deploy Preview	https://deploy-preview-120--h-test.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[DmitrySharabin]

@Zearin, would you be interested in testing our headless runner? As you might have noticed, I tested it on some of your tests. And the result leaves me very optimistic about the runner's future. 😊

[Zearin]

@DmitrySharabin I’ll give it a shot soon! 👍

[Zearin]

Hi @DmitrySharabin!

I followed your instructions (with an occasional mindless misstep 🤓). Here is the unabridged output:

Output:

amrogers@Anthonys-MacBook-Pro-16.local:
~/Projects/domrectified
192.168.1.159  npm install -D github:htest-dev/htest#headless-runner
npm warn gitignore-fallback No .npmignore file found, using .gitignore for file exclusion. Consider creating a .npmignore file to explicitly control published files.
changed 1 package, and audited 61 packages in 6s
26 packages are looking for funding

run npm fund for details
found 0 vulnerabilities
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npm install -D playwright
added 2 packages, and audited 63 packages in 11s
26 packages are looking for funding

run npm fund for details
found 0 vulnerabilities
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx playwright --help

Usage: npx playwright [options] [command]
Options:

-V, --version                          output the version number

-h, --help                             display help for command
Commands:

open [options] [url]                   open page in browser specified via -b, --browser

codegen [options] [url]                open page and generate code for user actions

install [options] [browser...]         ensure browsers necessary for this version of Playwright are

installed

uninstall [options]                    Removes browsers used by this installation of Playwright from the

system (chromium, firefox, webkit, ffmpeg). This does not include

branded channels.

install-deps [options] [browser...]    install dependencies necessary to run browsers (will ask for sudo

permissions)

cr [options] [url]                     open page in Chromium

ff [options] [url]                     open page in Firefox

wk [options] [url]                     open page in WebKit

screenshot [options]    capture a page screenshot

pdf [options]           save page as pdf

show-trace [options] [trace]           show trace viewer

test [options] [test-filter...]        run tests with Playwright Test

show-report [options] [report]         show HTML report

merge-reports [options] [dir]          merge multiple blob reports (for sharded tests) into a single

report

clear-cache [options]                  clears build and test caches

init-agents [options]                  Initialize repository agents

help [command]                         display help for command
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx playwright install --help

Usage: npx playwright install [options] [browser...]
ensure browsers necessary for this version of Playwright are installed
Options:

--with-deps   install system dependencies for browsers

--dry-run     do not execute installation, only print information

--list        prints list of browsers from all playwright installations

--force       force reinstall of already installed browsers

--only-shell  only install headless shell when installing chromium

--no-shell    do not install chromium headless shell

-h, --help    display help for command
Examples:


$ install

Install default browsers.


$ install chrome firefox

Install custom browsers, supports android, bidi-chromium, chrome, chrome-beta, chrome-for-testing, chromium, chromium-headless-shell, chromium-tip-of-tree, chromium-tip-of-tree-headless-shell, ffmpeg, firefox, firefox-beta, msedge, msedge-beta, msedge-dev, webkit, webkit-wsl.


amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx playwright install --list

Failed to install browsers

Error: ENOENT: no such file or directory, scandir '/Users/amrogers/Library/Caches/ms-playwright/.links'
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx playwright install webkit

Downloading WebKit 26.0 (playwright webkit v2248) from https://cdn.playwright.dev/dbazure/download/playwright/builds/webkit/2248/webkit-mac-15-arm64.zip

72.6 MiB [====================] 100% 0.0s

WebKit 26.0 (playwright webkit v2248) downloaded to /Users/amrogers/Library/Caches/ms-playwright/webkit-2248

Downloading FFmpeg (playwright ffmpeg v1011) from https://cdn.playwright.dev/dbazure/download/playwright/builds/ffmpeg/1011/ffmpeg-mac-arm64.zip

1 MiB [====================] 100% 0.0s

FFmpeg (playwright ffmpeg v1011) downloaded to /Users/amrogers/Library/Caches/ms-playwright/ffmpeg-1011
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx htest --headless --verbose --ci test/DOM

DOMRect-001/          DOMRect-001-issue.js  DOMRect-001.js        DOMRectList.js
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx htest --headless --verbose --ci test/DOMRect-0

DOMRect-001/          DOMRect-001-issue.js  DOMRect-001.js
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx htest --headless --verbose --ci test/DOMRect-001/

attributes.js   contructors.js  index.js
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx htest --headless --verbose --ci test/DOMRect-001/index.js

Launching headless runner...
Error importing environment headless browserType.launch: Executable doesn't exist at /Users/amrogers/Library/Caches/ms-playwright/chromium_headless_shell-1208/chrome-headless-shell-mac-arm64/chrome-headless-shell

╔═════════════════════════════════════════════════════════════════════════╗

║ Looks like Playwright Test or Playwright was just installed or updated. ║

║ Please run the following command to download new browsers:              ║

║                                                                         ║

║     npx playwright install                                              ║

║                                                                         ║

║ <3 Playwright Team                                                      ║

╚═════════════════════════════════════════════════════════════════════════╝

at Object.run (/Users/amrogers/Projects/domrectified/node_modules/htest.dev/src/env/headless.js:235:29) {

name: 'Error',

meta: {

name: 'Error',

message: "browserType.launch: Executable doesn't exist at /Users/amrogers/Library/Caches/ms-playwright/chromium_headless_shell-1208/chrome-headless-shell-mac-arm64/chrome-headless-shell\n" +

'╔═════════════════════════════════════════════════════════════════════════╗\n' +

'║ Looks like Playwright Test or Playwright was just installed or updated. ║\n' +

'║ Please run the following command to download new browsers:              ║\n' +

'║                                                                         ║\n' +

'║     npx playwright install                                              ║\n' +

'║                                                                         ║\n' +

'║ <3 Playwright Team                                                      ║\n' +

'╚═════════════════════════════════════════════════════════════════════════╝',

stack: "browserType.launch: Executable doesn't exist at /Users/amrogers/Library/Caches/ms-playwright/chromium_headless_shell-1208/chrome-headless-shell-mac-arm64/chrome-headless-shell\n" +

'╔═════════════════════════════════════════════════════════════════════════╗\n' +

'║ Looks like Playwright Test or Playwright was just installed or updated. ║\n' +

'║ Please run the following command to download new browsers:              ║\n' +

'║                                                                         ║\n' +

'║     npx playwright install                                              ║\n' +

'║                                                                         ║\n' +

'║ <3 Playwright Team                                                      ║\n' +

'╚═════════════════════════════════════════════════════════════════════════╝\n' +

'    at Object.run (/Users/amrogers/Projects/domrectified/node_modules/htest.dev/src/env/headless.js:235:29)'

}

}
amrogers@Anthonys-MacBook-Pro-16.local:

~/Projects/domrectified

192.168.1.159  npx htest --headless --browser webkit --verbose --ci test/DOMRect-001/index.js

Launching headless runner...
DOMRect-001 ✅ 6/6 PASS

└─┬  ✅ 6/6 PASS

└─┬ testSetAttributes ✅ 6/6 PASS

├─┬ x ✅ 3/3 PASS

│ ├──  PASS  "x" (0 ms)

│ ├──  PASS  "left" (0 ms)

│ └──  PASS  "right" (0 ms)

└─┬ y ✅ 3/3 PASS

├──  PASS  "y" (0 ms)

├──  PASS  "top" (0 ms)

└──  PASS  "bottom" (0 ms)