feat(e2e): add plugin sanity check test (RHIDP-13508) by gustavolira · Pull Request #4967 · redhat-developer/rhdh

gustavolira · 2026-06-17T20:08:41Z

Summary

Implements enhanced sanity checks for all enabled plugins in the RHDH deployment as part of RHIDP-13508.

Changes

New test file: e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts
- Validates all enabled packages from default.packages.yaml
- Checks package name format for all enabled plugins
- Validates disabled packages list is parseable
Integration: Added to SHOWCASE_SANITY_PLUGINS project in playwright.config.ts
Mock file: Created default.packages.yaml in repo root for local testing
- Real file is injected during CI deployment
- Mock allows local test execution

Test Behavior

The test runs in two parts:

Enabled packages validation: Reads all enabled packages and validates their format
Disabled packages validation: Ensures the disabled list is parseable

Currently validates package structure. Future enhancement can add actual plugin loading using @red-hat-developer-hub/cli-module-install-dynamic-plugins.

Testing

Local

yarn playwright test plugin-sanity-check --project showcase-sanity-plugins

CI

Test runs automatically in nightly showcase-sanity-plugins job where RHDH is deployed.

Jira

Implements: RHIDP-13508
Part of epic: RHIDP-13496 (Test Layer Infrastructure)

Checklist

Test created with proper component annotation
Integrated into existing Playwright project
ESM compatibility fixed (__dirname polyfill)
Mock file created for local testing
Test passes locally with mock data
Follows playwright-locators best practices
Follows ci-e2e-testing conventions

rhdh-qodo-merge · 2026-06-17T20:09:56Z

PR Summary by Qodo

feat(e2e): add Playwright plugin sanity-check test
🧪 Tests ✨ Enhancement ⚙️ Configuration changes 🕐 20-40 Minutes

Description

• Add Playwright sanity test that parses default.packages.yaml and validates plugin entries.
• Wire the new spec into the SHOWCASE_SANITY_PLUGINS Playwright project for nightly runs.
• Provide a mock default.packages.yaml in repo root to enable local execution.

Diagram

graph TD
  Config["playwright.config.ts"] --> Project["SHOWCASE_SANITY_PLUGINS"] --> Runner["Playwright runner"] --> Test["plugin-sanity-check.spec.ts"] --> Yaml[("default.packages.yaml")] --> Validate["Validate package entries"]
  CI["Nightly CI deployment"] --> Yaml

High-Level Assessment

The following are alternative approaches to this PR:

1. Resolve/install validation (require.resolve / dynamic import)

➕ Catches missing deps and broken package exports, not just naming issues
➕ Aligns better with the header comment about “can be loaded”
➖ May be flaky if the test environment doesn’t include all plugin packages
➖ Could increase runtime and require dependency management in CI images

2. Use dynamic plugin installer CLI in the test

➕ Validates the real dynamic-plugin acquisition path end-to-end
➕ Closer to how RHDH consumes plugins in deployment
➖ Adds infra complexity (network access, caching, time)
➖ Harder to keep deterministic in CI without pinning and mirrors

3. Schema/structure-only validation + stronger rules

➕ Keeps the test lightweight and deterministic
➕ Can still catch common config errors (missing fields, duplicates, invalid npm scope/name regex)
➖ Won’t detect runtime load failures or missing packages

Recommendation: The PR’s lightweight approach is reasonable for a nightly “sanity” gate, but the spec currently only checks package-name scoping (starts with @) and disabled-list structure. Consider either (a) tightening the structure validation (required fields, non-empty lists, duplicate detection, npm package-name regex) and adjusting the test header comment to match behavior, or (b) adding an optional require.resolve/import-based check when the environment guarantees plugin deps are present. Full CLI-based installation is higher fidelity but likely too heavy unless the team is ready to invest in caching and determinism.

Files changed (3) +149 / -0

Tests (1) +137 / -0

plugin-sanity-check.spec.tsAdd Playwright spec to validate enabled/disabled plugin entries +137/-0

Add Playwright spec to validate enabled/disabled plugin entries

• Introduces a new Playwright spec that reads 'default.packages.yaml', validates enabled plugin package name format (scoped), and asserts the disabled package list is parseable and well-formed. Adds a 'component=plugins' annotation and emits a simple console summary.

e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts

Other (2) +12 / -0

default.packages.yamlAdd mock default plugin package list for local runs +11/-0

Add mock default plugin package list for local runs

• Adds a repo-root 'default.packages.yaml' with enabled/disabled plugin entries to support local execution of the new Playwright sanity test. Notes that the real file is injected during CI deployment.

default.packages.yaml

playwright.config.tsInclude plugin sanity spec in SHOWCASE_SANITY_PLUGINS project +1/-0

Include plugin sanity spec in SHOWCASE_SANITY_PLUGINS project
• Extends the 'SHOWCASE_SANITY_PLUGINS' project 'testMatch' list to include 'plugin-sanity-check.spec.ts', ensuring it runs as part of the sanity-plugins CI job.
e2e-tests/playwright.config.ts

github-actions · 2026-06-17T20:10:18Z

The container image build workflow finished with status: cancelled.

codecov · 2026-06-17T20:13:23Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.77%. Comparing base (036635a) to head (988239c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4967      +/-   ##
==========================================
- Coverage   55.39%   54.77%   -0.62%     
==========================================
  Files         122      110      -12     
  Lines        2365     2147     -218     
  Branches      568      513      -55     
==========================================
- Hits         1310     1176     -134     
+ Misses       1048      970      -78     
+ Partials        7        1       -6

Flag	Coverage Δ
rhdh	`54.77% <ø> (-0.62%)`	⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 036635a...988239c. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

rhdh-qodo-merge · 2026-06-17T20:14:48Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

Context used

✅ Tickets: [RHIDP-13508](https://redhat.atlassian.net/browse/RHIDP-13508)

1. ~~No package resolution performed~~ ✓ Resolved 🐞 Bug ≡ Correctness

Description

The test "All enabled packages can be resolved" never resolves/imports any packages; it only checks
that the string starts with "@" and then records success. This can let CI pass even when a listed
plugin package is missing/unresolvable, making the new sanity check misleading.

Code

e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts[R44-83]

+  test("All enabled packages can be resolved", async () => {
+    // Read default.packages.yaml from rhdh repo root
+    const defaultPackagesPath = join(__dirname, "../../../default.packages.yaml");
+    const yamlContent = readFileSync(defaultPackagesPath, "utf8");
+    const config = yaml.parse(yamlContent) as DefaultPackagesConfig;
+
+    const enabledPackages = config.packages.enabled;
+    console.log(`\n📦 Testing ${enabledPackages.length} enabled packages...\n`);
+
+    const results: {
+      package: string;
+      status: "success" | "failed";
+      error?: string;
+    }[] = [];
+
+    for (const pkg of enabledPackages) {
+      const packageName = pkg.package;
+
+      try {
+        // Attempt to resolve the package
+        // Note: We can't actually import dynamic plugins here as they require
+        // a Backstage runtime, but we can at least verify the package name format
+        // and that it's listed in package.json dependencies
+
+        // Validate package name format
+        if (!packageName.startsWith("@")) {
+          throw new Error("Package name must be scoped (start with @)");
+        }
+
+        // For now, just verify the package is properly formatted
+        // Future enhancement: Use @red-hat-developer-hub/cli-module-install-dynamic-plugins
+        // to actually download and verify the plugins load
+
+        results.push({
+          package: packageName,
+          status: "success",
+        });
+
+        console.log(`✅ ${packageName}`);
+      } catch (error) {

Relevance

⭐⭐ Medium
Mixed history: team accepts improving E2E assertions/coverage (#3147,#4526) but rejected similar
“add runtime validation” checks (#4519).
PR-#3147
PR-#4526
PR-#4519

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The test’s core loop never calls any module-resolution API; it only validates a prefix and then
pushes a success result, so it cannot detect missing/unresolvable packages. Additionally, the E2E
test package itself does not declare the example plugin packages as dependencies, reinforcing that
the current implementation cannot be performing real resolution.

e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts[44-83]
e2e-tests/package.json[57-71]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The test name/comments say it verifies packages can be resolved/loaded, but the implementation only checks a naming convention (`startsWith('@')`) and always marks entries as `success` if that passes. This can silently miss real breakages (missing package, typo, unpublished artifact), because no `resolve`/`import` is ever attempted.

### Issue Context
- This is a Playwright E2E "sanity" test intended for CI signal quality. If the test claims resolvability but doesn’t actually do it, it provides misleading green runs.

### Fix Focus Areas
- e2e-tests/playwright/e2e/plugin-sanity-check.spec.ts[44-83]

### Suggested fix
Choose one (A is lowest-risk for now):

**A) Rename + adjust messaging to match reality**
- Rename the test to something like `All enabled packages have valid package identifiers`.
- Update the header comment and inline comments to remove "resolve/import" language.
- Update summary logs to reflect "validated format" rather than "loaded/resolved".

**B) Actually resolve packages (only if intended + feasible)**
- Implement a real resolution check (e.g., `createRequire(import.meta.url).resolve(packageName)` or Node’s `import.meta.resolve` where available) and fail on thrown errors.
- If doing this, ensure the packages being checked are expected to be present in the environment (dependencies or fetched artifacts), otherwise the test will become permanently red.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

github-actions · 2026-06-17T20:30:12Z

The container image build workflow finished with status: cancelled.

github-actions · 2026-06-17T20:37:16Z

The container image build workflow finished with status: cancelled.

github-actions · 2026-06-17T21:09:11Z

Image was built and published successfully. It is available at:

Implements review findings from PR redhat-developer#4967 code review: **Correctness fixes:** - Fix test.beforeAll signature to use test.info() (follows smoke-test.spec.ts pattern) - Remove unused manifestPath variable in plugin-dynamic-loading.spec.ts **Code organization:** - Extract DEFAULT_PACKAGES_PATH constant to avoid duplication - Move CONFIG_OVERRIDES to module scope for better reusability - Remove unused catalog-index-parser.ts (can be recreated when needed) **Type safety:** - Import and use BackendFeature type for require() calls instead of any - Improves type safety when loading plugin modules **Developer experience:** - Improve console.warn message to explain impact of missing _nodeModulePaths All changes are low-risk refactorings that improve code quality without changing behavior. Type checking passes with no errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

github-actions · 2026-06-18T13:01:56Z

The container image build workflow finished with status: cancelled.

github-actions · 2026-06-18T13:14:49Z

The container image build workflow finished with status: cancelled.

github-actions · 2026-06-18T13:17:06Z

The container image build workflow finished with status: cancelled.

github-actions · 2026-06-18T13:36:16Z

The container image build workflow finished with status: cancelled.

github-actions · 2026-06-18T13:39:51Z

The container image build workflow finished with status: cancelled.

github-actions · 2026-06-18T13:40:16Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-06-18T15:03:40Z

/test ?

gustavolira · 2026-06-18T15:04:09Z

/test e2e-ocp-helm-nightly

gustavolira · 2026-06-18T22:18:05Z

/test e2e-ocp-helm-nightly

github-actions · 2026-06-18T22:49:26Z

Image was built and published successfully. It is available at:

gustavolira · 2026-06-19T12:53:43Z

/test e2e-ocp-helm-nightly

github-actions · 2026-06-19T13:19:14Z

Image was built and published successfully. It is available at:

Implements review findings from PR redhat-developer#4967 code review: **Correctness fixes:** - Fix test.beforeAll signature to use test.info() (follows smoke-test.spec.ts pattern) - Remove unused manifestPath variable in plugin-dynamic-loading.spec.ts **Code organization:** - Extract DEFAULT_PACKAGES_PATH constant to avoid duplication - Move CONFIG_OVERRIDES to module scope for better reusability - Remove unused catalog-index-parser.ts (can be recreated when needed) **Type safety:** - Import and use BackendFeature type for require() calls instead of any - Improves type safety when loading plugin modules **Developer experience:** - Improve console.warn message to explain impact of missing _nodeModulePaths All changes are low-risk refactorings that improve code quality without changing behavior. Type checking passes with no errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

github-actions · 2026-06-19T19:42:46Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

github-actions · 2026-06-19T20:12:41Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-06-20T14:29:13Z

/test e2e-ocp-helm-nightly

Add explicit type annotation to require() call for better type safety. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Implements review findings from PR redhat-developer#4967 code review: **Correctness fixes:** - Fix test.beforeAll signature to use test.info() (follows smoke-test.spec.ts pattern) - Remove unused manifestPath variable in plugin-dynamic-loading.spec.ts **Code organization:** - Extract DEFAULT_PACKAGES_PATH constant to avoid duplication - Move CONFIG_OVERRIDES to module scope for better reusability - Remove unused catalog-index-parser.ts (can be recreated when needed) **Type safety:** - Import and use BackendFeature type for require() calls instead of any - Improves type safety when loading plugin modules **Developer experience:** - Improve console.warn message to explain impact of missing _nodeModulePaths All changes are low-risk refactorings that improve code quality without changing behavior. Type checking passes with no errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Based on final code review feedback: **Documentation improvements:** - Clarify default.packages.yaml is only used for local testing - Add comment explaining empty plugins config triggers catalog index extraction - Expand KNOWN_FAILURES documentation with reasons for each exclusion - Add JSDoc explaining purpose of KNOWN_FAILURES set **Why these changes:** - Makes it clearer when mock file vs CI-injected file is used - Documents non-obvious behavior (empty config → index extraction) - Helps future maintainers understand if failures can be re-enabled - All changes are documentation-only, no behavior changes Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Fixes all eslint errors reported in CI: **plugin-loader.ts:** - Change LoadedPlugin.feature type from any to BackendFeature - Fix eslint-disable comment for @typescript-eslint/no-require-imports - Rename CONFIG_OVERRIDES to configOverrides (camelCase) - Replace 'any' with 'unknown' in buildMergedConfig **plugin-dynamic-loading.spec.ts:** - Rename CORE_FEATURES to coreFeatures (camelCase) - Remove explicit 'any[]' type annotation (inferred correctly) **plugin-sanity-check.spec.ts:** - Add eslint-disable comments for __filename/__dirname (ESM compat) - Add eslint-disable comment for DEFAULT_PACKAGES_PATH (path convention) All changes improve type safety while maintaining functionality. Type checking and linting now pass with 0 errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

…lugin tests Improves documentation to address Qodo review feedback about test design. **Context:** Qodo flagged that plugin-sanity-check.spec.ts doesn't actually resolve packages, calling it a bug. This is intentional design - we have TWO complementary tests: 1. plugin-sanity-check.spec.ts (LIGHTWEIGHT ~seconds) - Fast format validation - Catches YAML/config errors - Does NOT download/load plugins 2. plugin-dynamic-loading.spec.ts (COMPREHENSIVE ~3 min) - Downloads from catalog index - Actually loads plugins with startTestBackend - Validates runtime behavior **Changes:** - Expanded JSDoc headers explaining the two-test architecture - Added inline comments explaining why format-only validation is intentional - Clarified what each test catches vs doesn't catch - Cross-referenced between the two test files This makes the design intent explicit and prevents future confusion about why plugin-sanity-check.spec.ts doesn't load plugins. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Applies Prettier formatting to plugin test files to pass CI checks. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Removes plugin-sanity-check.spec.ts (depends on default.packages.yaml which doesn't exist in RHDH repo) and keeps only plugin-dynamic-loading.spec.ts which reads from catalog index and validates all plugins. **CI Integration:** - Test runs in showcase-sanity-plugins namespace (already configured) - Uses PW_PROJECT_SHOWCASE_SANITY_PLUGINS Playwright project - Executed by run_sanity_plugins_check() in ocp-nightly.sh - Runtime: ~3 minutes for full plugin validation **What the test does:** 1. Downloads all plugins from CATALOG_INDEX_IMAGE via install-dynamic-plugins 2. Loads backend plugins and validates they have valid exports 3. Starts test backend with startTestBackend to verify plugins work 4. Validates frontend plugins have required bundle artifacts **Deployment:** - Namespace: showcase-sanity-plugins (fixed) - Values file: diff-values_showcase-sanity-plugins.yaml - Merged with base values during deployment - Uses same infrastructure as other showcase namespaces This completes RHIDP-13508 implementation - comprehensive plugin sanity check without cluster dependency, running in nightly CI. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

…c-loading test Adds @backstage/plugin-catalog-backend and @backstage/plugin-scaffolder-backend as devDependencies to resolve import errors in plugin-dynamic-loading.spec.ts. **Root Cause:** The test imports catalogPlugin and scaffolderPlugin to use as core features when starting the test backend with startTestBackend(), but these packages were not declared in e2e-tests/package.json, causing: Error: Cannot find package '@backstage/plugin-catalog-backend' **Dependencies Added:** - @backstage/plugin-catalog-backend@3.5.0 (matches packages/backend version) - @backstage/plugin-scaffolder-backend@3.3.0 (matches packages/backend version) **Verification:** - TypeScript compilation passes - yarn install completes successfully - Versions match those used in packages/backend/package.json for consistency This fixes the CI failure in e2e-ocp-helm job where the showcase namespace tests failed due to missing dependencies. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Adds ESM compatibility polyfill for __dirname in plugin-dynamic-loading.spec.ts to fix ReferenceError when calling patchModuleResolution(). **Error:** ReferenceError: __dirname is not defined in ES module scope at patchModuleResolution(join(__dirname, "..", "..", "node_modules")) **Fix:** - Import fileURLToPath and dirname from Node.js path/url modules - Define __filename and __dirname polyfills at module scope - Add eslint-disable comments for naming-convention (matches pattern from plugin-sanity-check.spec.ts) This is the standard ESM compatibility pattern used across e2e tests. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Adds plugin-dynamic-loading.spec.ts to testIgnore for the showcase project to prevent it from running in PR checks (e2e-ocp-helm job). **Problem:** The test was running in the showcase project during e2e-ocp-helm PR checks, but it requires: 1. CATALOG_INDEX_IMAGE environment variable 2. install-dynamic-plugins CLI to download plugins 3. ~3 minutes execution time 4. Cluster deployment (not available in showcase namespace setup) **Why this test shouldn't run in showcase:** - showcase project runs in e2e-ocp-helm (PR checks) - This test is designed for showcase-sanity-plugins (nightly only) - PR checks should be fast (<10 min), this test takes ~3 min alone - Test requires specific deployment setup not present in showcase **Where the test SHOULD run:** - showcase-sanity-plugins project (configured in playwright.config.ts) - Executed by run_sanity_plugins_check() in ocp-nightly.sh - Only runs in e2e-ocp-helm-nightly job (nightly, not PR checks) This ensures the test only runs in the appropriate environment where the infrastructure supports plugin extraction and validation. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Changes stdio from 'pipe' to 'inherit' and adds detailed error handling to show the actual failure reason when install-dynamic-plugins CLI fails. **Previous Issue:** `stdio: 'pipe'` was hiding all stderr/stdout, making it impossible to diagnose why the CLI was failing. Error was just: Error: Command failed: npx @red-hat-developer-hub/cli-module-install-dynamic-plugins **Fix:** 1. Changed `stdio: 'inherit'` to show real-time output during execution 2. Added try-catch with detailed error message including: - Exit code - Catalog index image being used - Stdout/stderr (if available) - Original error message **Why this helps:** - Reveals the actual error (network issue, missing dependency, permission, etc.) - Shows progress during the ~3 minute download - Makes CI debugging possible **Note:** The CI Dockerfile already has skopeo installed (line 82-84), so the failure is NOT due to missing container runtime. The real cause will now be visible in the CI logs. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Adds --version check before running install-dynamic-plugins to verify the CLI is available, and improves error messages to guide debugging. **Changes:** 1. Check CLI availability with --version before running install command 2. Print command and env vars before execution for transparency 3. Simplify error message since stdio='inherit' shows output in logs above **Next run will show:** - ✓ CLI version (proves package is accessible) - Command being executed - CATALOG_INDEX_IMAGE value - Real-time output during execution (stdio='inherit') - Clear error pointing to logs above on failure This will help diagnose whether the failure is: - CLI not found (npx issue) - CLI found but crashes (will see error in logs) - Network/registry issue (will see in logs) - Permission/skopeo issue (will see in logs) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

The install-dynamic-plugins CLI requires the 'install' subcommand: npx @red-hat-developer-hub/cli-module-install-dynamic-plugins install <dir> Without it, the CLI was using the default mode which doesn't match our usage pattern with CATALOG_INDEX_IMAGE environment variable. This should resolve the exit code 1 failure in CI.

Split monolithic plugin-loader.ts into focused modules following POC structure: Created: - plugin-types.ts: Type definitions (PluginEntry, LoadedPlugin, PluginError, etc.) - plugin-config.ts: Configuration (KNOWN_FAILURES, buildMergedConfig) - plugin-reporter.ts: Reporting functions (consistent logging across test phases) Refactored: - plugin-loader.ts: Pure loading logic (loadManifest, loadBackendPlugins, validateFrontendBundle) - plugin-dynamic-loading.spec.ts: Uses reporter for all console output Benefits: - Better separation of concerns - Consistent reporting format - Easier to test and maintain - Matches POC structure from PR redhat-developer#4523 All files pass TypeScript, ESLint, and Prettier checks.

Improvements based on code review: 1. Catalog index fallback: Fail explicitly in CI if CATALOG_INDEX_IMAGE is unset - Prevents tests from silently using outdated fallback version - Adds clear error message for CI misconfiguration 2. Test credentials documentation: Add comment clarifying dummy values - Makes it clear config overrides are for validation only - Prevents confusion about real credential requirements 3. Module resolution robustness: Add NODE_PATH fallback - Handles case where Node.js internal API is unavailable - Platform-aware path separator (Windows vs Unix) - More resilient to future Node.js changes 4. Frontend bundle validation clarity: Improve comments and error messages - Explains modern vs legacy bundle formats - More descriptive error messages 5. Entry point resolution: Better error diagnostics - Shows package.json main field in error message - Easier to debug plugin packaging issues 6. Race condition documentation: Add comment about serial execution - Documents assumption that showcase-sanity-plugins runs serially - Clarifies module patch safety 7. Timeout explanation: Document why 5 minutes was chosen - Breaks down: 3min download + 2s startup + 2min buffer All changes improve code clarity, robustness, and maintainability.

Address Qodo review feedback on test hermeticity: - Skip test when CATALOG_INDEX_IMAGE is not set in local development - Provides clear skip message directing developers to opt-in explicitly - In CI, the test always runs (env var is always set) - Prevents non-hermetic test runs in local environments - Aligns with existing e2e test skip patterns in the codebase This makes the test opt-in for local development while ensuring it always runs in CI with the correct catalog version. Refs: Qodo review comment #4781427938

The previous logic had a bug where CI environment with missing env var would skip the early skip check but then throw an error later. Fixed by: 1. Check CI + missing env var first (fail-fast) 2. Then check local + missing env var (skip test) 3. Removed redundant check after assignment 4. Removed non-null assertion (! operator) - no longer needed This ensures: - CI always fails loudly if env var is missing - Local development skips gracefully if env var is missing - No unreachable code paths

Changed from throwing error to skipping test when CATALOG_INDEX_IMAGE is missing, regardless of CI environment. Rationale: - Test is already excluded from PR checks via testIgnore in playwright.config.ts - Only runs in nightly showcase-sanity-plugins where env var is set - If it somehow runs in an environment without the env var, skip gracefully instead of failing with confusing error - Better UX: skip with clear message vs cryptic environment error This aligns with Playwright best practices for environment-dependent tests.

github-actions · 2026-06-26T13:13:15Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

gustavolira · 2026-06-26T13:17:58Z

/test e2e-ocp-helm-nightly

The plugin-dynamic-loading.spec.ts test was being skipped in nightly jobs because CATALOG_INDEX_IMAGE was not set by default. This commit sets a branch-aware default value: - main → quay.io/rhdh/plugin-catalog-index:next - release-X.Y → quay.io/rhdh/plugin-catalog-index:X.Y The default can still be overridden via Gangway for RC/GA testing. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

sonarqubecloud · 2026-06-26T15:16:56Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

gustavolira · 2026-06-26T15:42:54Z

/test e2e-ocp-helm-nightly

github-actions · 2026-06-26T15:45:31Z

Image was built and published successfully. It is available at:

openshift-ci · 2026-06-26T16:41:12Z

@gustavolira: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ocp-helm-nightly	`988239c`	link	false	`/test e2e-ocp-helm-nightly`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci · 2026-06-26T16:41:20Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci Bot requested review from rm3l and subhashkhileri June 17, 2026 20:08

rhdh-qodo-merge Bot added Tests Enhancement labels Jun 17, 2026

gustavolira force-pushed the main branch from 6628116 to 1ef2bc2 Compare June 19, 2026 19:42

gustavolira force-pushed the main branch from 165931c to a8d76ac Compare June 23, 2026 13:41

gustavolira and others added 18 commits June 26, 2026 10:12

refactor(e2e): improve type annotation in plugin-loader

8291d2a

Add explicit type annotation to require() call for better type safety. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

style(e2e): fix prettier formatting

2c0e2f5

Applies Prettier formatting to plugin test files to pass CI checks. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

gustavolira force-pushed the main branch from dd363cf to ca0ee5a Compare June 26, 2026 13:12

openshift-ci Bot removed the needs-rebase label Jun 26, 2026

gustavolira mentioned this pull request Jun 26, 2026

feat(poc): Docker-free in-process dynamic-plugin smoke harness redhat-developer/rhdh-plugin-export-overlays#2714

Open

openshift-ci Bot added the needs-rebase label Jun 26, 2026

Uh oh!

Conversation

gustavolira commented Jun 17, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test Behavior

Testing

Local

CI

Jira

Checklist

Uh oh!

rhdh-qodo-merge Bot commented Jun 17, 2026

PR Summary by Qodo

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

codecov Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rhdh-qodo-merge Bot commented Jun 17, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

gustavolira commented Jun 18, 2026

Uh oh!

gustavolira commented Jun 18, 2026

Uh oh!

gustavolira commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

gustavolira commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

gustavolira commented Jun 20, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

gustavolira commented Jun 26, 2026

Uh oh!

sonarqubecloud Bot commented Jun 26, 2026

Quality Gate passed

Uh oh!

gustavolira commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

openshift-ci Bot commented Jun 26, 2026

Uh oh!

openshift-ci Bot commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

gustavolira commented Jun 17, 2026 •

edited by openshift-ci Bot

Loading

codecov Bot commented Jun 17, 2026 •

edited

Loading

rhdh-qodo-merge Bot commented Jun 17, 2026 •

edited by openshift-ci Bot

Loading