Auto-balancing Playwright test shards

[aslushnikov]

Hey folks,

I used to work on Playwright, and I noticed that Swup uses Playwright too, including sharding in CI.

These days I’m building Flakiness.io: a GitHub-native test analytics dashboard that tracks flaky tests and detects regressions in pull requests.

We’re prototyping Playwright shard balancing, where Flakiness.io uses historical test timings to produce better-balanced shards. Since Swup has a real-world Playwright setup, I’d be very happy if you’re open to testing it on your workload.

OSS projects can use Flakiness.io for free. I’d be happy to help set it up and see if you find it useful.

[daun]

Sounds interesting! We're not adding many features to the library these days, but we'll definitely take a look once we're moving toward a new major version where testing becomes important again.

Am I understand this right that Flakiness would help determine the correct shard balance for best test performance?

[aslushnikov]

Yes, the idea is to use historical test durations data to balance shards. It should work very nice in theory.

As a side bonus, you'd also get a cloud-hosted dashboard with all your tests - so you'd be able to drop the blob reports / merge-reports dance you currently do to upload html report as an artifact. It looks like this: https://flakiness.io/flakiness/vitest