feat(e2e): cluster-free local E2E harness (legacy app, Tier B)

[gustavolira]

Summary

Adds a cluster-free local E2E harness for the legacy frontend (packages/app, Tier B): run real Playwright E2E against RHDH without an OpenShift/Kubernetes cluster or container images. Playwright boots the backend and the legacy app dev server in-process and drives the browser against them — and it runs on GitHub Actions in a no-cluster phase.

Part of RHIDP-13501 (E2E Test Optimization), Layer 4a spike RHIDP-15075. Builds on the PoC in #4523 and the backend dynamic-plugin loader from RHIDP-13508.

How it runs

.github/workflows/e2e-cluster-free.yaml (cluster-free GitHub Actions job):

1. Install deps + skopeo
2. Populate dynamic-plugins-root from the public catalog index via the install-dynamic-plugins CLI (same mechanism as the nightly sanity check)
3. yarn e2e:legacy-local — boots backend + legacy app dev servers in-process, runs the verified test

Locally:

CATALOG_INDEX_IMAGE=quay.io/rhdh/plugin-catalog-index:latest \
  npx @red-hat-developer-hub/cli-module-install-dynamic-plugins install dynamic-plugins-root
yarn --cwd e2e-tests e2e:legacy-local

Why

The existing E2E specs target the legacy app, so this harness runs them unmodified — no rewrite. Dynamic frontend plugins load via Scalprum exactly as in-cluster. This is faster and lighter than rhdh-local (container-image based) for automated testing.

What's verified

• Legacy app + backend boot off-cluster; dynamic frontend plugins load via Scalprum.
• The production RHDH home page (Quick Access from the dynamic home-page plugin) renders off-cluster, and the existing guest-signin-happy-path home-page test passes unmodified. The default run is grep-scoped to exactly that test.
• A globalSetup fails fast with the populate command if dynamic-plugins-root is empty.

Known issues / follow-ups (documented in docs/e2e-tests/local-e2e-harness.md)

• global-header plugin mounting still needs config sorting; specs that navigate via the top-right profile dropdown depend on it, so the run is scoped to the verified home-page test for now. Widen testMatch/grep as specs are validated.
• CI scope starts at e2e-tests/** + app-config*.yaml; can widen to packages/app/** / packages/backend/** once proven stable.
• app-next harness: not included — packages/app-next can't load dynamic frontend plugins yet (blocked upstream on standard Module Federation assets). Tracked under RHIDP-15082.

🤖 Generated with Claude Code

[rhdh-qodo-merge]

PR Summary by Qodo

Add cluster-free local Playwright E2E harnesses (legacy Tier B + app-next)
✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Description

• Add Playwright harnesses that boot backend + frontend dev servers in-process, off-cluster.
• Provide legacy Tier B harness to run existing specs unchanged with Scalprum dynamic plugins.
• Document setup, limitations, and add guest-auth + in-memory SQLite config overlay.

Diagram

graph TD
  PW(["Playwright runner"]) --> CH{"Harness"} -->|"Legacy Tier B"| LEG(["Legacy app dev server (janus-cli)"]) --> BR["Browser-driven E2E"]
  CH -->|"app-next"| NEXT(["App-next dev server (backstage-cli)"]) --> BR
  PW --> BE(["Backend dev server (backstage-cli)"]) --> BR
  CFG[["app-config.local-e2e.yaml"]] --> BE
  CFG --> LEG
  CFG --> NEXT
  DPR[("dynamic-plugins-root")] --> LEG
  subgraph Legend
    direction LR
    _svc(["Dev server"]) ~~~ _file[["Config file"]] ~~~ _db[("Plugins root")] ~~~ _dec{"Decision"} ~~~ _box["Test target"]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Single Playwright config with projects (legacy + app-next)

• ➕ Eliminates duplication of reporters/timeouts/webServer wiring
• ➕ Allows running one or both harnesses via Playwright project selection
• ➕ Centralizes env handling (PATH, CI reuseExistingServer)
• ➖ Slightly more complex conditional webServer setup
• ➖ Harder to keep legacy testMatch constraints isolated without care

2. Use container-based rhdh-local for local E2E

• ➕ Closest to production image runtime; fewer dev-server quirks
• ➕ Avoids needing local workspace CLIs on PATH
• ➖ Requires container runtime and image pulls
• ➖ Heavier/slower for automated E2E; contradicts ‘cluster-free/in-process’ goal

Recommendation: Keep the PR’s approach (in-process dev servers via Playwright webServer) because it directly optimizes for fast, cluster-free automated E2E and preserves existing legacy specs unchanged. If this becomes long-lived, consider consolidating common Playwright config into a shared base and using Playwright projects to reduce config duplication while keeping distinct legacy vs app-next server commands.

Files changed (6) +381 / -1

Tests (3) +248 / -0

playwright.app-next-local.config.tsAdd Playwright config to boot backend + app-next dev server +94/-0

Add Playwright config to boot backend + app-next dev server

• Defines a Playwright configuration that starts the backend and packages/app-next dev server via webServer, reusing servers locally and forcing fresh starts in CI. Ensures repo-root .bin CLIs are discoverable via PATH and sets reporting/timeout defaults for the harness.

e2e-tests/playwright.app-next-local.config.ts

playwright.legacy-local.config.tsAdd Tier B Playwright config for legacy app with dynamic plugins +105/-0

Add Tier B Playwright config for legacy app with dynamic plugins

• Creates a legacy harness config that runs a curated subset of existing specs off-cluster. Boots backend and packages/app via backstage-cli/janus-cli with layered configs including dynamic plugins and the local E2E overlay, and reuses servers outside CI.

e2e-tests/playwright.legacy-local.config.ts

guest-identity.spec.tsAdd app-next local guest sign-in + identity smoke test +49/-0

Add app-next local guest sign-in + identity smoke test

• Introduces a minimal E2E spec validating the app-next harness: guest sign-in succeeds, core navigation is present, and the Settings page renders the guest Backstage identity. Assertions intentionally match app-next UI behavior rather than legacy home-page expectations.

e2e-tests/playwright/app-next-local/guest-identity.spec.ts

Documentation (1) +105 / -0

local-e2e-harness.mdDocument cluster-free local E2E harness usage and constraints +105/-0

Document cluster-free local E2E harness usage and constraints

• Adds a dedicated guide describing the legacy Tier B and app-next harnesses, dynamic-plugins-root population paths, and what’s currently verified. Documents key limitations (global-header mounting, app-next dynamic plugin loading, and external-service-dependent specs) and explains why this approach differs from rhdh-local.

docs/e2e-tests/local-e2e-harness.md

Other (2) +28 / -1

app-config.local-e2e.yamlAdd local E2E config overlay (guest auth + in-memory DB) +25/-0

Add local E2E config overlay (guest auth + in-memory DB)

• Introduces a shared config overlay for cluster-free E2E runs. Enables the guest auth provider (including CI-safe allowance) and forces an in-memory better-sqlite3 backend database for fully self-contained runs.

app-config.local-e2e.yaml

package.jsonAdd yarn scripts to run legacy-local and app-next-local harnesses +3/-1

Add yarn scripts to run legacy-local and app-next-local harnesses

• Adds two Playwright entrypoint scripts for the new harness configs. Keeps existing tooling scripts intact while enabling one-command local harness execution.

e2e-tests/package.json

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Wrong server reused 🐞 Bug ☼ Reliability

Description

With reuseExistingServer enabled outside CI, Playwright will reuse any process responding on the
configured localhost URLs, even if it's not the intended RHDH harness. Because the URLs are
hard-coded to localhost:3000 and localhost:7007, this can lead to tests running against
stale/foreign servers and producing misleading results.

Code

e2e-tests/playwright.legacy-local.config.ts[R88-100]

+      url: backendReadiness,
+      reuseExistingServer: !process.env.CI,
+      timeout: 180 * 1000,
+      stdout: "pipe",
+      stderr: "pipe",
+    },
+    {
+      command: `janus-cli package start ${sharedConfigArgs}`,
+      cwd: "../packages/app",
+      env: { ...process.env, PATH: `${repoRootBin}:${process.env.PATH}` },
+      url: frontendUrl,
+      reuseExistingServer: !process.env.CI,
+      timeout: 240 * 1000,

Relevance

⭐⭐ Medium

No prior review evidence on reuseExistingServer risks; pattern likely tolerated for local dev but
could be tightened.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Both configs hard-code the target URLs/ports and enable reuse of any existing server when CI is
not set, which makes it possible to run tests against an unintended process bound to those ports.

e2e-tests/playwright.legacy-local.config.ts[31-33]
e2e-tests/playwright.legacy-local.config.ts[88-100]
e2e-tests/playwright.app-next-local.config.ts[24-26]
e2e-tests/playwright.app-next-local.config.ts[75-89]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The harness sets `reuseExistingServer: !process.env.CI` while targeting fixed `http://localhost:3000` and `http://localhost:7007/...`. If anything else is listening on those ports (or a previous run is still alive), Playwright may attach to the wrong process.

### Issue Context
This affects both `playwright.legacy-local.config.ts` and `playwright.app-next-local.config.ts`.

### Fix
Pick one:
1) Make reuse explicit (opt-in), e.g. `reuseExistingServer: process.env.REUSE_EXISTING_SERVER === 'true'`.
2) Use per-run ports (random/free port) and pass them into backend/frontend via config/env overrides, and derive `baseURL`/readiness URLs from those ports.
3) Add a lightweight identity check before reusing (e.g., fetch a known endpoint and validate a header/body) and fail if it doesn’t match.

### Fix Focus Areas
- e2e-tests/playwright.legacy-local.config.ts[31-33]
- e2e-tests/playwright.legacy-local.config.ts[88-100]
- e2e-tests/playwright.app-next-local.config.ts[24-26]
- e2e-tests/playwright.app-next-local.config.ts[75-89]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. NODE_OPTIONS clobbered 🐞 Bug ☼ Reliability

Description

The local harness webServer env sets NODE_OPTIONS to a fixed value, overwriting any pre-existing
NODE_OPTIONS from the environment. This can break harness startup in CI/dev setups that rely on
other Node flags (e.g., memory limits or preload hooks).

Code

e2e-tests/playwright.legacy-local.config.ts[R83-87]

+      env: {
+        ...process.env,
+        PATH: `${repoRootBin}:${process.env.PATH}`,
+        NODE_OPTIONS: "--no-node-snapshot",
+      },

Relevance

⭐ Low

Repo already sets NODE_OPTIONS="--no-node-snapshot" unconditionally in Docker/CI; clobbering appears
accepted.

PR-#2297

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Both harness configs spread ...process.env but then unconditionally override NODE_OPTIONS, which
discards any upstream flags.

e2e-tests/playwright.legacy-local.config.ts[83-87]
e2e-tests/playwright.app-next-local.config.ts[70-74]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The Playwright `webServer.env` blocks replace `NODE_OPTIONS` with `"--no-node-snapshot"`, which drops any existing `NODE_OPTIONS` settings inherited from the parent environment.

### Issue Context
This affects both the legacy and app-next local harness configs.

### Fix
Construct `NODE_OPTIONS` by appending `--no-node-snapshot` to any existing value, e.g.:

```ts
const nodeOptions = [process.env.NODE_OPTIONS, '--no-node-snapshot']
 .filter(Boolean)
 .join(' ');

NODE_OPTIONS: nodeOptions,
```

### Fix Focus Areas
- e2e-tests/playwright.legacy-local.config.ts[83-87]
- e2e-tests/playwright.app-next-local.config.ts[70-74]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-bfbe9dc6

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-ebddd2cc

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.39%. Comparing base (b5b37c4) to head (f2019bb).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5005   +/-   ##
=======================================
  Coverage   55.39%   55.39%           
=======================================
  Files         122      122           
  Lines        2365     2365           
  Branches      568      563    -5     
=======================================
  Hits         1310     1310           
- Misses       1048     1049    +1     
+ Partials        7        6    -1     

Flag	Coverage Δ
rhdh	55.39% <ø> (ø)

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b5b37c4...f2019bb. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-1bf02b56

[gustavolira]

/test e2e-ocp-helm-nightly

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-ab72af06

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-f2019bbc

[openshift-ci]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[openshift-ci]

@gustavolira: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ocp-helm-nightly	1bf02b5	link	false	/test e2e-ocp-helm-nightly
ci/prow/e2e-ocp-helm	39ab02f	link	true	/test e2e-ocp-helm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.