Stabilize nightly Playwright failures across environments - #798
Stabilize nightly Playwright failures across environments#798sofiaIvarsRitense wants to merge 5 commits into
Conversation
📝 WalkthroughSummary by CodeRabbit
WalkthroughUpdated Playwright CI video retention to keep recordings on failure. Enhanced access-control role creation to support manual role entry across modal variants. Scoped dropdown and modal interactions, stabilized case-management forms and toggles, updated selectors and widget paths, hardened shared test cleanup, and improved task-assignment polling. 🚥 Pre-merge checks | ✅ 2✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Ran locally, everything clean, and got two errors: Match type is visible for text with non-dropdown field type at case-details-management-search-fields/page.ts:137 135 | await expect(option).toBeVisible({timeout: 2_000});
Feature 2 — Cases (User) › 2.6 — Execute tasks within case Locator: locator('valtimo-case-detail-task-list').locator('cds-clickable-tile').filter({ has: locator('span.task__title:has-text("Change name")') }).first() Call log:
219 | // Completing the "Change name" task removes its tile from the list. The case's
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4a282a12-4fd1-4bb5-9045-e45f155db8d6
📒 Files selected for processing (10)
e2e/tests/case-details-management-form-flows/case-details-management-form-flows.spec.tse2e/tests/case-details-management-form-flows/page.tse2e/tests/case-details-management-tabs/case-details-management-tabs.spec.tse2e/tests/case-details-management-tabs/page.tse2e/tests/case-details-management-tasks/case-details-management-tasks.spec.tse2e/tests/case-details-management-tasks/page.tse2e/tests/case-details-management-widgets/case-details-management-widgets.spec.tse2e/tests/case-details-management-widgets/case-details-management-widgets.tse2e/tests/case-details-management-widgets/page.tse2e/tests/iko/iko-search-field/page.ts
| test.afterAll(async () => { | ||
| await formFlowsPage.deleteFormFlowViaApi(formFlowTestData.key); | ||
| await context.close(); | ||
| if (context) await context.close(); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
files=(
"e2e/tests/case-details-management-form-flows/case-details-management-form-flows.spec.ts"
"e2e/tests/case-details-management-tabs/case-details-management-tabs.spec.ts"
"e2e/tests/case-details-management-tasks/case-details-management-tasks.spec.ts"
"e2e/tests/case-details-management-widgets/case-details-management-widgets.spec.ts"
)
for f in "${files[@]}"; do
echo "===== $f ====="
wc -l "$f"
sed -n '1,140p' "$f" | cat -n
doneRepository: valtimo-platform/valtimo
Length of output: 25648
🌐 Web query:
Playwright Test class fixtures afterAll beforeAll hooks still run after teardown API failures context.close in finally
💡 Result:
In Playwright Test, the test runner is designed to execute all applicable hooks, including afterEach and afterAll, even if a test or a preceding hook fails [1][2][3]. This ensures that resource cleanup can proceed as intended regardless of test outcomes [2]. Regarding your query on teardown failures and context.close: 1. Hook Execution on Failure: Playwright continues to run all applicable hooks even if some of them encounter errors [1][3]. When a test fails, the worker process is typically discarded and restarted for subsequent tests [1][4]. Despite this, Playwright ensures that the necessary after hooks are triggered before the worker is torn down [2]. 2. Finally Blocks and Context.close: If you are using a try-finally block to manage browser context lifecycle, ensure that all asynchronous operations on the context or its pages (e.g., page actions, network requests) have fully completed before reaching the finally block [5]. A common cause of Target closed or similar errors is calling context.close while asynchronous operations are still in flight, or using the context/page after it has already been closed by a previous cleanup step [5]. 3. Worker Restart Behavior: Note that because Playwright may restart the worker process upon test failures, resources initialized in beforeAll or as worker-scoped fixtures will be re-initialized in the new worker process [1][4]. If your teardown logic in afterAll relies on state that was corrupted or lost during a failure, it may behave unexpectedly, but the hook itself will still be invoked by the runner [1][2]. For robust cleanup, ensure your afterAll and fixture teardown logic is resilient to errors, and verify that any manual context.close calls are only made after all operations on that context have settled [5].
Citations:
- 1: https://playwright.dev/docs/api/class-test
- 2: fix(test runner): ensure we run after hooks after failures microsoft/playwright#8102
- 3: https://github.com/microsoft/playwright/blob/d3d436d3/docs/src/test-api/class-test.md
- 4: https://github.com/microsoft/playwright/blob/main/docs/src/test-retries-js.md
- 5: https://screenshotrun.com/blog/playwright-target-page-context-or-browser-has-been-closed
🌐 Web query:
Playwright Test afterAll after failed beforeAll page object null context cleanup behavior
💡 Result:
In Playwright Test, the behavior of afterAll hooks and worker cleanup following a failure is designed to ensure test isolation and system stability. When a test fails, Playwright intentionally discards the entire worker process, including its browser context and pages, to prevent state leakage [1][2]. Consequently, if a beforeAll hook fails or a test fails during execution, Playwright will not necessarily run the afterAll hook in the original, now-discarded worker process; instead, it restarts the worker for the next batch of tests [3][2]. Key behaviors to understand: 1. Worker Restart: When a failure occurs, Playwright tears down the current worker process [2]. Any resources (like a manually created page or context) initialized in a beforeAll hook that rely on that worker's lifecycle are effectively destroyed by the process exit [1][2]. 2. Hook Execution: Playwright is designed to run all applicable beforeAll and afterAll hooks even if some fail [3][4]. However, the primary mechanism for cleanup in Playwright is worker teardown, not afterAll hooks [5]. 3. Page Object Nullability: If you attempt to access a page object in an afterAll hook that was initialized in a beforeAll that failed (or if the afterAll runs in a new, clean worker process where that specific variable was never initialized), you may encounter null reference errors. Recommendation: The recommended way to handle shared resources is to use worker-scoped fixtures instead of beforeAll/afterAll hooks [6][5]. Worker-scoped fixtures manage their own setup and teardown lifecycle automatically, even in the event of failures, and provide a typesafe way to clean up resources without manually managing global variables that might be null [7][5]. If you must use beforeAll, ensure your afterAll is defensive (e.g., checking if the object exists before calling .close()) [5][8].
Citations:
- 1: [BUG] - test.AfterAll is being executed after each failed test instead of after all tests microsoft/playwright#26632
- 2: https://playwright.dev/docs/test-retries
- 3: https://playwright.dev/docs/api/class-test
- 4: [BUG] Failure in beforeAll does not end test when there are multiple beforeAll blocks microsoft/playwright#28285
- 5: https://runebook.dev/en/docs/playwright/api/class-test/test-after-all
- 6: [QUESTION]: The first cases for each suite have failed when running reusable page structure in POM microsoft/playwright#32819
- 7: https://playwright.dev/docs/test-fixtures
- 8: https://dev.to/aktibaba/pom-as-fixture-and-the-beforeall-trap-playwright-typescript-ch8-50d3
🌐 Web query:
Playwright Test github issue afterAll runs after beforeAll fails afterAll not runs after beforeAll fails
💡 Result:
In Playwright Test, the execution behavior of afterAll hooks following a failed beforeAll hook—or failures in general—is a deliberate design choice related to how Playwright handles worker processes and test isolation [1][2][3]. Key Technical Points: 1. Worker Process Lifecycle: Playwright Test isolates tests by worker processes [4][5]. If a test fails, Playwright discards the entire worker process, including the browser instance, to ensure a clean environment for subsequent tests [4][6][3]. 2. afterAll Execution: Because the worker process is terminated upon failure, the afterAll hook is executed as part of the teardown for that specific worker [1][3]. If multiple tests are running and a failure occurs, the current worker's afterAll hook will run, and a new worker process will be started for the remaining tests [4][7][3]. Consequently, you may observe the afterAll hook running multiple times throughout a test suite execution [3]. 3. beforeAll Failure Behavior: Playwright is designed to execute all registered beforeAll hooks, even if one of them fails [2][8]. If a beforeAll hook fails, Playwright will still attempt to run any remaining beforeAll hooks before skipping the actual tests in that suite [2][8]. 4. Intended Design: This behavior is consistent and documented by the Playwright team as "working as intended" [2][3]. It is designed to prevent "poisoned" processes from affecting subsequent tests and to ensure that teardown (afterAll) runs for the resources created within that specific worker's lifecycle [1][6][3]. If you need hooks to run strictly once or manage resources globally, Playwright documentation recommends using Global Setup/Teardown or Project Dependencies/Teardown features rather than relying solely on file-level beforeAll/afterAll hooks [7]. Top results: [1][2][7][3]
Citations:
- 1: https://playwright.dev/docs/api/class-test
- 2: [BUG] Failure in beforeAll does not end test when there are multiple beforeAll blocks microsoft/playwright#28285
- 3: [BUG] - test.AfterAll is being executed after each failed test instead of after all tests microsoft/playwright#26632
- 4: https://github.com/microsoft/playwright/blob/main/docs/src/test-retries-js.md
- 5: [Bug]: With playwright test, an 'expect' fail causes beforeAll to re-run microsoft/playwright#7554
- 6: [BUG]Playwright test.beforeAll hook executes more than once in a test suite if any test fails/throw error microsoft/playwright#8472
- 7: [Question] is there any way to make beforeall and afterall is only executed once even if a test block fails microsoft/playwright#23331
- 8: [Feature]: skip remaining before* hooks if one of them fails microsoft/playwright#33314
🌐 Web query:
"run afterAll in that same worker process" "beforeAll" "Playwright Test"
💡 Result:
In Playwright Test, the afterAll hook is designed to run in the same worker process as the beforeAll hook, provided that no test failures occur [1][2]. Playwright automatically manages this lifecycle: when all tests within a suite or file pass, the worker process remains alive, executing the beforeAll hook first, followed by the tests, and finally the afterAll hook in sequence [2][3]. However, if any test fails, Playwright Test considers the worker process compromised and will discard it [2]. In this scenario, the afterAll hook will still run in that specific worker process before it is shut down, but any subsequent tests or retries will be executed in a completely new worker process, which will then trigger its own beforeAll and afterAll hooks [1][2]. Key points regarding the worker lifecycle: - Consistency: Playwright intentionally keeps the worker process alive to run both hooks to ensure proper setup and teardown, as long as the process remains healthy [2][3]. - Failures: Upon failure, the current worker is terminated to prevent state leakage or flaky behavior from affecting further tests [2]. The afterAll hook is executed as part of the worker's cleanup sequence before the process exits [4]. - Parallelism: If you are using parallel mode or separate test files, each worker operates independently [5][6]. You cannot rely on sharing state between them; each worker process handles its own beforeAll and afterAll hooks for the tests assigned to it [5][6]. - Global Hooks: If you need to run code before or after every test file without re-declaring them in every file, you can use worker-scoped fixtures instead of beforeAll/afterAll [3][7].
Citations:
- 1: https://playwright.dev/docs/api/class-test
- 2: https://playwright.dev/docs/test-retries
- 3: https://playwright.dev/docs/test-fixtures
- 4: https://github.com/microsoft/playwright/blob/54e92be7/packages/playwright/src/worker/workerMain.ts
- 5: https://github.com/microsoft/playwright/blob/main/docs/src/test-parallel-js.md
- 6: https://playwright.dev/docs/test-parallel
- 7: https://github.com/microsoft/playwright/blob/main/docs/src/test-fixtures-js.md
🌐 Web query:
Playwright Test afterAll after first beforeAll fails within describe workerMain
💡 Result:
In Playwright Test, if a beforeAll hook fails, the following behavior occurs regarding afterAll and other hooks: 1. Hook Execution: Playwright is designed to run all beforeAll and afterAll hooks, even if some of them fail [1][2]. If you have multiple beforeAll hooks, a failure in the first one will not prevent the remaining beforeAll hooks from executing [3]. 2. Test Execution: If a beforeAll hook fails, all tests within that describe block or test file are skipped [4]. 3. afterAll Execution: Even if a beforeAll hook fails, any defined afterAll hooks will still execute [4]. This ensures that any necessary teardown or cleanup logic defined in your afterAll hooks is performed [4]. It is important to note that when a test or hook fails, Playwright typically discards the current worker process and starts a new one for subsequent tests [5][6]. Because the afterAll hook is part of the lifecycle of the worker that encountered the failure, it runs in that same worker process before it is discarded [1][5]. If you have retries enabled, these hooks will run again in the new worker process created for the retry attempt [7][5]. This behavior is intentional as it guarantees that teardown logic is consistently applied, even in scenarios where setup fails [3][7]. If you have dependencies where a later hook should not run if an earlier one fails, you should handle that logic within the hook functions themselves (e.g., using status checks or conditional logic) [8].
Citations:
- 1: https://playwright.dev/docs/api/class-test
- 2: https://github.com/microsoft/playwright/blob/d3d436d3/docs/src/test-api/class-test.md
- 3: [BUG] Failure in beforeAll does not end test when there are multiple beforeAll blocks microsoft/playwright#28285
- 4: https://microsoft-playwright.mintlify.app/test-runner/test-hooks
- 5: https://playwright.dev/docs/next/test-retries
- 6: https://playwright.dev/docs/test-retries
- 7: [Question] afterAll hooks and test failures microsoft/playwright#26683
- 8: [Feature]: skip remaining before* hooks if one of them fails microsoft/playwright#33314
🌐 Web query:
Playwright Test workerMain beforeAll failures afterAll call stack afterAll teardown partial fixtures
💡 Result:
In Playwright Test, worker behavior upon failure is highly specific to ensure a clean environment for subsequent tests [1][2]. When a test fails, Playwright discards the current worker process, which triggers the teardown of all worker-scoped fixtures and the execution of any relevant afterAll hooks within that worker [3][1][4]. Key aspects of this execution flow include: Worker Process Isolation and Failure: Playwright restarts the worker process after a test failure [5][2][4]. This is an intentional design to prevent state pollution from a failed test [2]. Because the worker process is being torn down, Playwright ensures that afterAll hooks are executed before the worker process fully exits [3][1]. BeforeAll Failure: If a beforeAll hook fails, the entire suite is generally considered failed, and Playwright will skip the remaining tests in that suite [3][6]. In this scenario, afterAll hooks are still executed for any suites that have already successfully run their beforeAll hooks, ensuring proper resource cleanup [3]. Worker Fixture Teardown: Worker-scoped fixtures are torn down when the worker process shuts down [7]. If a failure occurs, the teardown for these fixtures is triggered as part of the worker shutdown process [3]. Playwright is designed to continue running these cleanup tasks even if some individual hooks fail [5][8]. Call Stack and Partial Fixtures: Because Playwright manages fixture lifecycles via a runner (workerMain.ts and _fixtureRunner), failures inside beforeAll or specific tests can lead to partial fixture setups if multiple fixtures are involved [3][6]. If an error occurs, Playwright makes a best-effort attempt to tear down fixtures that were successfully initialized [3]. If you need to distinguish between successful and failed runs within your afterAll or teardown logic (e.g., to skip certain cleanup tasks on failure), a common workaround is to use test.afterEach() to track test status via test.info().status and store that information in a variable accessible to afterAll() [9]. Alternatively, for global teardown tasks that should only run if all tests pass, project dependencies with a defined teardown property in playwright.config.ts are the recommended approach [10]. Top results: [3], [1], [4]
Citations:
- 1: https://playwright.dev/docs/test-retries
- 2: https://playwright.dev/docs/test-parallel
- 3: https://github.com/microsoft/playwright/blob/c0cc9802/packages/playwright/src/worker/workerMain.ts
- 4: [BUG] - test.AfterAll is being executed after each failed test instead of after all tests microsoft/playwright#26632
- 5: https://playwright.dev/docs/api/class-test
- 6: https://github.com/microsoft/playwright/blob/5790370e/packages/playwright/src/worker/workerMain.ts
- 7: https://playwright.dev/docs/test-fixtures
- 8: https://github.com/microsoft/playwright/blob/54e92be7/docs/src/test-api/class-test.md
- 9: How NOT to run afterAll on failure? microsoft/playwright#27197
- 10: https://stackoverflow.com/questions/78218922/playwright-teardown-only-if-all-specs-have-passed
🌐 Web query:
Playwright Test "beforeAll" cleanup "workerMain" afterAll throws if beforeAll throws
💡 Result:
In Playwright Test, the execution behavior when a beforeAll hook throws an error is governed by a design choice to ensure robustness and cleanup [1][2]. Key behaviors include: 1. All Hooks Attempt to Run: Playwright will continue to execute all applicable beforeAll hooks for a suite even if one of them fails [1][2]. It does not stop immediately upon the first failure in a beforeAll hook; it attempts to run all defined hooks in that category [2]. 2. Test Skipping: If any beforeAll hook throws an error, the tests within that suite are marked as failed (or skipped) because the necessary setup could not be completed [3][4]. 3. Cleanup and afterAll: Playwright is designed to perform cleanup reliably. Even if a beforeAll hook fails, the framework attempts to run the corresponding afterAll hooks to ensure resources are properly disposed of [3][5]. The worker process responsible for the failed setup is typically discarded afterward, as Playwright treats a failed setup or a failed test as a reason to restart the worker process to maintain a clean environment for subsequent tests [1][6][7]. 4. Worker Main Logic: Internally, the workerMain.ts logic is designed to catch errors during these hook executions, store the first encountered error, and ensure that fixture teardown scopes are executed for each hook, regardless of whether that specific hook succeeded or failed [3][4][5]. This ensures that even if a beforeAll hook throws, the teardown logic is triggered to prevent resource leaks [5][8]. In summary, a failure in beforeAll does not prevent the execution of subsequent beforeAll hooks or the eventual afterAll cleanup hooks; rather, it triggers a cascade where the suite's tests are skipped, the cleanup is performed, and the worker process is typically recycled [1][5][2].
Citations:
- 1: https://playwright.dev/docs/api/class-test
- 2: [BUG] Failure in beforeAll does not end test when there are multiple beforeAll blocks microsoft/playwright#28285
- 3: https://github.com/microsoft/playwright/blob/303901d7/packages/playwright/src/worker/workerMain.ts
- 4: https://github.com/microsoft/playwright/blob/54e92be7/packages/playwright/src/worker/workerMain.ts
- 5: https://github.com/microsoft/playwright/blob/4d289016/packages/playwright/src/worker/workerMain.ts
- 6: https://playwright.dev/docs/test-retries
- 7: [BUG] - test.AfterAll is being executed after each failed test instead of after all tests microsoft/playwright#26632
- 8: https://github.com/microsoft/playwright/blob/c0cc9802/packages/playwright/src/worker/workerMain.ts
🌐 Web query:
Google Groups "@playwright/test after beforeAll fails afterAll not called" "worker"
💡 Result:
In Playwright Test, worker processes are intentionally shut down after a test failure to guarantee a clean, pristine environment for subsequent tests [1][2][3]. Because the entire worker process is discarded, any afterAll hook associated with that specific worker process will not execute for the failed test, and any beforeAll hooks required by subsequent tests will need to be re-run in a newly created worker process [4][1][5]. This behavior is fundamental to Playwright's architecture and cannot currently be disabled [6]. The primary implications are: 1. Lifecycle Hook Re-execution: Because a new worker is spawned to continue testing, beforeAll hooks are re-executed in the new process [7][8][1]. 2. Incomplete Cleanup: Since the original worker is terminated immediately upon failure, the afterAll hook in that specific worker does not run for the failed test, which can lead to issues if you are managing external resources that require explicit cleanup [9][3]. If you are encountering issues due to this behavior, common community-suggested workarounds include: - Using test.describe.configure({ mode: 'serial' }): This ensures tests in a group run sequentially in the same worker, and if one test fails, subsequent tests in that group are skipped rather than triggering a worker restart for each [1][2]. - Implementing Global Setup/Teardown: For resources that must persist or be cleaned up exactly once regardless of test failure, consider using Playwright's global setup and teardown configuration rather than local beforeAll/afterAll hooks [8][3]. - External Resource Management: If managing external dependencies, ensure your cleanup logic is robust enough to handle instances where afterAll is not called, or use an external tool to manage resource lifecycles [9][3].
Citations:
- 1: https://playwright.dev/docs/test-retries
- 2: https://playwright.dev/docs/test-parallel
- 3: [Feature]: An option to NOT shutdown worker after test failure microsoft/playwright#32803
- 4: https://playwright.dev/docs/api/class-test
- 5: https://github.com/microsoft/playwright/blob/d3d436d3/docs/src/test-api/class-test.md
- 6: [Question] Failed soft assertions are retriggering beforeAll hook within test.describe group microsoft/playwright#17662
- 7: [Bug]: With playwright test, an 'expect' fail causes beforeAll to re-run microsoft/playwright#7554
- 8: Allow BeforeAll To Be Called Just Once in a Describe in FullyParallel Mode microsoft/playwright#28201
- 9: [Question]: Why isn't
test.afterAllran after a failed test? microsoft/playwright#8005
🌐 Web query:
GitHub "Playwright" "runAfterAllHooks" "beforeAll failed"
💡 Result:
In Playwright, if a beforeAll hook fails, the test runner will continue to execute all remaining beforeAll hooks as well as all afterAll hooks [1][2][3]. The Playwright test runner is designed to be resilient by ensuring that teardown logic, such as afterAll hooks, is executed even if setup logic (beforeAll) encounters errors [4][3]. Key behaviors include: 1. Hook Execution Resilience: Playwright continues to run all applicable hooks even if some of them fail [1][5]. This ensures that resources can still be cleaned up properly [4]. 2. Test Execution: If a beforeAll hook fails, the tests associated with that hook will be skipped [6][3]. However, because the runner continues to execute subsequent beforeAll hooks, any logic contained within them will still run, even if the initial test suite was already effectively aborted due to the earlier failure [2][6]. 3. Worker Behavior: On test failures, the worker process is often restarted, and hooks may run again in the new worker process [1][5]. This behavior is intentional. While some users have requested features to skip remaining beforeAll hooks upon a failure to avoid unnecessary execution [6], Playwright maintains the current design to ensure consistent cleanup [4]. To avoid issues where subsequent beforeAll hooks depend on the success of a previous one, you can combine logic into a single beforeAll block or implement conditional checks within your hooks [6].
Citations:
- 1: https://playwright.dev/docs/api/class-test
- 2: [BUG] Failure in beforeAll does not end test when there are multiple beforeAll blocks microsoft/playwright#28285
- 3: https://github.com/microsoft/playwright/blob/c0cc9802/tests/playwright-test/hooks.spec.ts
- 4: fix(test runner): ensure we run after hooks after failures microsoft/playwright#8102
- 5: https://github.com/microsoft/playwright/blob/54e92be7/docs/src/test-api/class-test.md
- 6: [Feature]: skip remaining before* hooks if one of them fails microsoft/playwright#33314
🌐 Web query:
"afterAll" "beforeAll" "failed" "Playwright Test" "run the afterAll hooks"
💡 Result:
In Playwright Test, the afterAll hooks are executed even if a preceding beforeAll hook fails [1][2]. Playwright is designed to ensure that teardown logic within hooks runs as reliably as possible to prevent resource leaks [2]. Key details regarding this behavior: 1. Guarantee of Execution: Playwright attempts to run all applicable afterAll hooks, regardless of whether tests passed, failed, or if there were failures in the beforeAll or other setup hooks [3][1][2]. 2. Worker Lifecycle: If a failure occurs during execution, the worker process is often restarted for subsequent tests to ensure a clean state [3]. Before this restart, Playwright will still attempt to run the afterAll hooks to allow for necessary cleanup [2]. 3. Multiple Hooks: If you have multiple hooks defined, Playwright will continue to execute them even if one of them fails [3][4]. It captures the first error encountered but does not stop the execution of subsequent hooks [2]. This behavior ensures that resources (such as database connections or browser contexts) initialized in beforeAll are consistently torn down, minimizing side effects when tests or setup processes fail [1][2].
Citations:
- 1: https://microsoft-playwright.mintlify.app/test-runner/test-hooks
- 2: https://github.com/microsoft/playwright/blob/c0cc9802/packages/playwright/src/worker/workerMain.ts
- 3: https://playwright.dev/docs/api/class-test
- 4: [BUG] Failure in beforeAll does not end test when there are multiple beforeAll blocks microsoft/playwright#28285
🌐 Web query:
site:github.com/microsoft/playwright "runAfterAllHooks" "beforeAll"
💡 Result:
There is no standard hook or API method named runAfterAllHooks in major test frameworks like Playwright. It appears you may be conflating standard lifecycle hook terminology with custom logic or internal runner behavior. In Playwright, beforeAll is a standard lifecycle hook that executes once per worker process before all tests in a file or describe block [1][2]. Key behavioral points regarding these hooks in Playwright include: 1. Execution Order: When multiple beforeAll hooks are defined, they are executed in the order of their registration [1]. Similarly, multiple afterAll hooks are also executed in the order of their registration [3]. 2. Persistence During Failures: Playwright is designed to ensure that afterAll hooks run even if tests or other hooks fail [4][5]. By design, Playwright generally attempts to execute all applicable beforeAll and afterAll hooks, even if one of them fails [6][7]. 3. Worker Lifecycle: beforeAll and afterAll hooks run at the worker level [8]. If a worker process restarts (e.g., due to test failures or configuration), these hooks will run again in the new worker process [1][3]. If you are encountering a situation where you need specific cleanup logic to run after all hooks, it is standard practice to use the test.afterAll hook, which is explicitly designed to handle teardown after all tests and other setup hooks have concluded [1][3]. If your intent was to find a way to sequence or force-run specific code, ensure you are utilizing the test.afterAll lifecycle method rather than looking for a runner-specific command.
Citations:
- 1: https://github.com/microsoft/playwright/blob/54e92be7/docs/src/test-api/class-test.md
- 2: https://github.com/microsoft/playwright/blob/d3d436d3/docs/src/test-fixtures-js.md
- 3: https://github.com/microsoft/playwright/blob/d3d436d3/docs/src/test-api/class-test.md
- 4: fix(test runner): ensure we run after hooks after failures microsoft/playwright#8102
- 5: [Question] afterAll hooks and test failures microsoft/playwright#26683
- 6: [BUG] Failure in beforeAll does not end test when there are multiple beforeAll blocks microsoft/playwright#28285
- 7: [Feature]: skip remaining before* hooks if one of them fails microsoft/playwright#33314
- 8: [Bug]: Section of fixture after "use" is executed before the test method microsoft/playwright#34803
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "===== workerMain references in checked files ====="
rg -n "runAfter|beforeAll|afterAll|fixture.*teardown|fixtureScope" packages e2e . --glob '!node_modules' --glob '!dist' --glob '!build' | head -200 || true
echo
echo "===== package files mentioning playwright test version ====="
for f in package.json e2e/package.json pnpm-lock.yaml package-lock.json yarn.lock; do
[ -f "$f" ] && { echo "--- $f"; rg -n '"`@playwright/test`"|playwright' "$f" | head -80; }
doneRepository: valtimo-platform/valtimo
Length of output: 23073
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in e2e/tests/case-details-management-form-flows/case-details-management-form-flows.spec.ts \
e2e/tests/case-details-management-tabs/case-details-management-tabs.spec.ts \
e2e/tests/case-details-management-tasks/case-details-management-tasks.spec.ts \
e2e/tests/case-details-management-widgets/case-details-management-widgets.spec.ts; do
echo "===== $f ====="
sed -n '1,180p' "$f" | cat -n
doneRepository: valtimo-platform/valtimo
Length of output: 30433
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "===== e2e beforeAll afterAll teardown-related patterns ====="
rg -n "test\.afterAll|test\.beforeAll|afterEach\(.*test\.info|setup.*success|successFlag|context\.close|page.*close" e2e/tests | head -250
echo
echo "===== focused beforeAll/afterAll sections for teardown patterns ====="
python3 - <<'PY'
from pathlib import Path
for path in sorted(Path('e2e/tests').rglob('*.spec.ts')):
txt=path.read_text(errors='ignore')
if 'afterAll' in txt and ('context.close' in txt or 'page.close' in txt):
# print relevant counts only
start=max(0, txt.find('test.beforeAll')); end=min(len(txt), txt.find('test.describe(', txt.find('7' if False else '')))
print(path, 'before_all_count', txt.count('test.beforeAll'), 'after_all_count', txt.count('test.afterAll'), 'with_guarded_page_context_in_after', ('context.close' in txt or 'page.close' in txt) and ('if ' in txt))
PYRepository: valtimo-platform/valtimo
Length of output: 18945
Mark setup status before cleanup and close the context in finally.
These hooks use the manually constructed context / page / widgetTabKey without recording whether beforeAll completed, so a partial setup can throw in afterAll before context.close() from the guarded branch. Put shared context closure in a finally block and condition cleanup on a setup-success flag for each file.
e2e/tests/case-details-management-form-flows/case-details-management-form-flows.spec.tse2e/tests/case-details-management-tabs/case-details-management-tabs.spec.tse2e/tests/case-details-management-tasks/case-details-management-tasks.spec.tse2e/tests/case-details-management-widgets/case-details-management-widgets.spec.ts
📍 Affects 4 files
e2e/tests/case-details-management-form-flows/case-details-management-form-flows.spec.ts#L52-L54(this comment)e2e/tests/case-details-management-tabs/case-details-management-tabs.spec.ts#L59-L70e2e/tests/case-details-management-tasks/case-details-management-tasks.spec.ts#L59-L65e2e/tests/case-details-management-widgets/case-details-management-widgets.spec.ts#L64-L91
Source: MCP tools
| private async fillTabForm(title: string, key: string) { | ||
| await expect(async () => { | ||
| await this.tabNameInput.fill(title); | ||
| await this.tabKeyInput.fill(key); | ||
| await expect(this.tabNameInput).toHaveValue(title); | ||
| await expect(this.tabKeyInput).toHaveValue(key); | ||
| }).toPass({timeout: 10_000}); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Wait for Carbon’s deferred focus before filling the form.
toPass() returns immediately on its first successful attempt; it does not wait through the documented ~100 ms focus callback. That callback can still steal focus after this helper returns, leaving the key incomplete and recreating the flaky disabled-Create path.
Proposed fix
await this.page.getByRole('button', {name: 'Widgets component'}).click();
await expect(this.addTabConfirmButton).toBeVisible();
+ await expect(this.tabNameInput).toBeFocused();
await this.fillTabForm(title, key);| const textFields = [ | ||
| ...(column.title ? [{input: this.columnTitleInput, value: column.title}] : []), | ||
| {input: this.columnKeyInput, value: column.key}, | ||
| ]; | ||
|
|
||
| await this.fillTextFields(textFields); | ||
| await this.fillValuePathManually(this.columnPathToggle, this.columnPathInput, column.path); | ||
| await this.selectDropdownItem(this.columnDisplayTypeDropdown, column.displayType); | ||
| // A reset that lands after the first fill would have cleared these again, so | ||
| // re-verify (and re-fill) right before relying on the form being valid. | ||
| await this.fillTextFields(textFields); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Synchronize with the deferred reset before relying on required fields.
toPass() exits on an immediate success, so the 240 ms reset may run afterward. The final retry restores only title/key; if the reset clears the manually entered path, the form remains invalid.
e2e/tests/case-details-management-tasks/page.ts#L224-L234: wait for the pending reset before filling, or include the manual path in the final reconciliation.e2e/tests/case-details-management-tasks/page.ts#L256-L274: apply the same full-field reconciliation for search fields.
📍 Affects 1 file
e2e/tests/case-details-management-tasks/page.ts#L224-L234(this comment)e2e/tests/case-details-management-tasks/page.ts#L256-L274
2 participants
Describe the changes
Link to the related Github issue: https://github.com/generiekzaakafhandelcomponent/atlas-internal/issues/536
Specify the code branch location: story/536-stabilize-flaky-e2e-tests
Relevant comments:
Breaking changes
Documentation
New features or changes that have been introduced have been documented.
Tests
Unit tests have been added that cover these changes
Integration tests have been added that cover these changes
Describe the testing steps
Security
The Secure by Design principle has been applied to these changes
Added or changed REST API endpoints have authentication and authorization in place
Valtimo access control checks have been implemented
Dependencies
Newly added dependencies do not introduce known vulnerabilities/CVE's and are in line with the Valtimo license