ci: restructure e2e test workflows and add admin test API#472
Open
ci: restructure e2e test workflows and add admin test API#472
Conversation
PR Environment Deployment FailedThe PR environment deployment encountered an error. Please check the workflow logs for details. Common issues:
|
PR Environment Deployment FailedThe PR environment deployment encountered an error. Please check the workflow logs for details. Common issues:
|
Same fix as OTP polling: exponential backoff (500ms -> 4s cap) instead of linear 500ms retries.
Stop forwarding error.message to the client in 500 responses. The detail is still logged server-side via console.error.
Admin routes now return "Internal server error" instead of forwarding error.message. Update test expectations to match.
deploy-keeperhub.yaml is now a reusable workflow called via workflow_call from ci-pipeline.yml, not triggered by workflow_run.
…ction Deduplicate ~40 lines of setup steps repeated across 6 jobs. New .github/actions/setup-node-pnpm/action.yml with two boolean inputs: install-playwright and discover-plugins.
- Rename "Local" context to "Ephemeral" to match naming convention - Fix mermaid diagram: replace non-existent check-e2e-label with check-labels - Update e2e-tests-ephemeral.yml trigger to reflect workflow_call pattern - Correct remote vitest test count from 114+ to ~130 - Add composite action (setup-node-pnpm) documentation to workflow docs
…ignals Add data attributes to app components for test automation: - workflow-canvas: data-ready for canvas load state - org-switcher: data-state for switching/loading/ready - accept-invite: data-page-state for hydration state Replace waitForTimeout/networkidle with element assertions: - auth.setup: remove 2s sleeps, use domcontentloaded + org-switcher wait - auth utils: replace networkidle with org-switcher visibility - workflow utils: use data-ready, waitForURL, element assertions - invitations: replace retry loop with data-page-state wait - workflow.test/schedule-trigger.test: remove all waitForTimeout calls - organization-wallet.test: replace toast race with element assertion Stabilize playwright config: - fullyParallel: false, workers: 1 (serial to avoid shared-state conflicts) - retries: 2 (handles environmental flakiness) - reporter: github + html in CI, list locally
- Unskip analytics-gas, scheduled-workflow, web3-balance, para-wallet tests - Remove stop-execution.test.ts and ORG-4 placeholder (unimplemented UI) - Fix Para Wallet "Create Wallet" ambiguous selector (data-slot scoping) - Fix analytics-gas scrollIntoView race (wait for attachment first) - Remove CI sharding from e2e-tests-ephemeral.yml (serial execution) - Move regex literal to top-level scope (biome lint fix) - Use consistent button[role="combobox"] selector in signIn() - Update docs with CI execution model and stability decisions
🚀 PR Environment DeployedYour PR environment has been successfully deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
Read the test API key from AWS Parameter Store instead of GitHub Secrets so the CI runner uses the same value as the deployed app.
🚀 PR Environment DeployedYour PR environment has been successfully deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
Read TEST_API_KEY from the deployed pod environment so the CI runner uses the exact same value as the app. Skip ephemeral and vitest-remote jobs on this branch to iterate faster on playwright-remote.
When a synchronize event only changes files under .github/, tests/, or config files like playwright.config/vitest.config, skip the deploy step and just re-run the tests against the existing deployment.
🚀 PR Environment DeployedYour PR environment has been successfully deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
🚀 PR Environment DeployedYour PR environment has been successfully deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
The deployed PR environment runs NODE_ENV=development, not test/CI, so rate limiting was still active. Add DISABLE_AUTH_RATE_LIMIT env var to PR environment values and check it in the auth config.
Use better-auth customRules to bypass rate limiting when requests include a valid X-Test-API-Key header. Playwright config sends this header automatically when TEST_API_KEY env var is set. This keeps rate limiting active for real users while allowing E2E tests to run without hitting limits in PR, staging, and prod environments.
🚀 PR Environment DeployedYour PR environment has been successfully deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
Add testFetch() and getTestHeaders() to vitest E2E utils for future tests that hit auth endpoints. Also add X-Test-API-Key to Playwright admin-fetch headers.
Fix customRules return type (return currentRule instead of undefined), remove unused biome-ignore suppression, drop unnecessary async.
🚀 PR Environment DeployedYour PR environment has been successfully deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
Remote and ephemeral E2E test jobs are disabled (if: false) across deploy-keeperhub, deploy-pr-environment, and e2e-tests-ephemeral workflows while auth/rate-limit infrastructure is being stabilised.
Remote tests gated by ENABLE_E2E_REMOTE_TESTS, ephemeral tests by ENABLE_E2E_EPHEMERAL_TESTS. Both are GitHub repository variables. Currently ephemeral=true, remote=false.
🚀 PR Environment DeployedYour PR environment has been successfully deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
🚀 PR Environment DeployedYour PR environment has been successfully deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
workflow_runcascade that caused phantom CI runs on every PR pushworkflow_callreusable workflows coordinated by orchestrator files (ci-pipeline.yml,release-pipeline.yml)e2e-tests.ymltoe2e-tests-ephemeral.ymlwith ephemeral/remote naming convention/api/admin/test/otp,/api/admin/test/invitation) for remote Playwright DB lookups.github/actions/setup-node-pnpmcomposite actionChanges
CI/CD
workflow_runchains toworkflow_callorchestratorsactions/checkoutto v6 across all workflow filesinstall-playwrightanddiscover-pluginsinputsTest Reliability
signOut()now throws on missing user menu instead of silently passingerror.messageAdmin Test API
GET /api/admin/test/otp?email=-- latest OTP lookupGET /api/admin/test/invitation?email=-- pending invitation lookupTEST_API_KEYwith timing-safe comparison@techops.servicesdomainDocumentation
workflow_runreferences, mermaid diagrams, test countsTest plan
pnpm test:e2e:vitestpasses locallypnpm test:e2e(Playwright) passes locally