Skip to content

fix: three-layer test architecture and the defects it surfaced#728

Open
robsongajunior wants to merge 44 commits into
devfrom
test/webkit
Open

fix: three-layer test architecture and the defects it surfaced#728
robsongajunior wants to merge 44 commits into
devfrom
test/webkit

Conversation

@robsongajunior

@robsongajunior robsongajunior commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

The webkit testing initiative: a three-layer test architecture for @aziontech/webkit (functional browser-mode suite, Storybook visual regression, one ordered CI pipeline), plus the component and packaging defects the new suites surfaced. 36 commits.

Layer 1 — Functional suite (Vitest browser mode)

  • Vitest 4 + @vitest/browser-playwright running every test in real headless Chromium — behavior, structure, and ARIA on a real DOM, not jsdom.
  • 75 suites / 1,588 tests (1,582 passing, 6 documented skips) covering every exported component, including the eight that had no suite: popover, accordion, split-button, scroll-area, input-group, field-input-group, field-phone-number, field-text-switch. The templates/ components are intentionally untested — the folder is slated for deletion.
  • Shared foundation:
    • vitest.config.ts — browser mode, CI retry, v8 coverage config (report-only for now)
    • src/test/setup.ts — anchor-navigation guard (real <a href> clicks would tear down the test iframe) + per-test cleanup
    • src/test/axe.ts — axe-core structural a11y helper; every suite asserts zero violations (visual rules like color-contrast are disabled because this env doesn't run Tailwind — pixels belong to Layer 2)
  • Adapted to the post-merge dev: the standardized stories tree (per-component folders, renamed exports) and the breaking event refactor — suites now assert the positional (event, item) payloads of dropdown select, split-button item-click, breadcrumb navigate, pick-list item-double-click and table row-click, including the originating MouseEvent/KeyboardEvent.
  • Conventions documented in .claude/rules/testing.md and enforced by a new enforce-test-exists.mjs hook; the component-scaffold skill now emits a test file.

Real component defects documented via it.skip (verified red, not faked green)

  1. split-buttonDropdown.Trigger (role="button" span) wraps the real IconButton <button> → axe nested-interactive (split-button.vue:141).
  2. input-grouparia-required is set on the role="group" root, where it is not an allowed ARIA attribute → axe aria-allowed-attr.
  3. field-phone-number — the country trigger (role="combobox") has no accessible name: the aria-label lands on the Select wrapper div (field-phone-number.vue:143). Blocks two axe assertions (same root cause).
  4. field-phone-number — the dial code is passed as <SelectTrigger> slot content, but select-trigger.vue has no default slot → the trigger shows the raw ISO code ("BR" instead of "+55") (field-phone-number.vue:146).
  5. breadcrumb — selecting an overflow Dropdown option doesn't surface the navigate emit under synthetic interaction; needs a component-level check.

Layer 2 — Visual regression (Storybook test-runner)

  • @storybook/test-runner + jest-image-snapshot: every story (75 files / 309 snapshots) screenshotted at #storybook-root (1280×720, fonts ready, animations at 1ms via the visual-test-freeze style, pointer parked) and compared against committed baselines with a 0.01% threshold.
  • 309 linux baselines committed — regenerated post-merge, so they match the standardized story IDs and the new parameters.layout geometry. Baselines are platform-specific: linux/ is the CI contract; darwin/ supports local runs and stays gitignored.
  • Templates excluded from the visual layer (test-runner-jest.config.jstestPathIgnorePatterns): the stories/templates/ files are never collected — no render smoke, no snapshot, no baseline on -u. Their orphan baselines were removed.
  • playwright pinned to ^1.61.1 in both packages/webkit and apps/storybook, so the CLI that installs Chromium in CI is the same version the runners use.
  • Scripts: test:visual / test:visual:static / test:visual:ci / test:visual:update in the storybook app; at the root, storybook:test:visual* plus storybook:test:visual:update:baseline (dispatches the regeneration workflow).

Layer 3 — CI: one ordered pipeline, no loose checks

  • governance.yml now carries the whole chain:
    changes → security ∥ lint ∥ types → build ∥ storybook ∥ tests → visual → Governance Gate
    • tests — the Vitest browser suite, sharded 4×, gated on lint + types (stories are unit fixtures, so apps/storybook changes retrigger it).
    • visual — last link: builds Storybook and runs test:visual:ci against the committed baselines only after everything upstream is green; uploads __diff_output__ on failure.
    • The gate's results map now includes tests and visual; cleanly-skipped jobs still pass (path-filtered).
  • package-webkit-test.yml removed — its shard matrix moved into governance, so the suite no longer double-runs per PR.
  • app-storybook-generate-baseline.yml — dispatch-only baseline tool: run with update_baselines=true, download the visual-baselines-linux artifact, commit its contents.

Defects surfaced and fixed by the suites

Packaging / publish hygiene

  • The npm tarball no longer ships tests: files negations (!src/**/*.test.{ts,js}, !src/test/**) plus a pack:check guard script that fails if a test file leaks into npm pack.
  • pnpm-workspace.yaml: @swc/core postinstall disabled (the native binary ships via the platform optionalDep; the script is only a download fallback).
  • Storybook imports @aziontech/webkit directly — the @aziontech/webkit.dev Vite alias is gone (follow-up to the [NO-ISSUE] fix: resolve prettier format and vue-tsc type-check CI failures #726 package rename). Dead webkit-v3 ref and unused root deps removed.

How to run

pnpm webkit:test                          # full browser-mode suite (root)
pnpm webkit:test:coverage                 # + v8 coverage report
pnpm storybook:test:visual                # build storybook + visual run against local baselines
pnpm storybook:test:visual:update:baseline  # dispatch the CI workflow that regenerates linux baselines

Verification

  • Functional suite: 75 files, 1,582 passed / 6 skipped (skips = the documented component defects above)
  • Visual suite: 75 story files / 309 linux baselines, templates excluded by config
  • vue-tsc --noEmit, ESLint (--max-warnings 0), and prettier all clean

robsongajunior and others added 15 commits July 7, 2026 15:10
Vitest browser mode (Playwright Chromium) + @testing-library/vue +
composeStories, axe-core a11y helper, theme CSS loaded in setup so
contrast checks are real, publish-safe files negation (no test files in
the tarball), sharded CI workflow, and root passthrough scripts.

Adopts the direction validated by the #704 proposal; hardens the gaps
(publish leak, unstyled-DOM a11y, pure-module coverage, CI scale).
…onfig

Functional Button coverage (polymorphism, click emission, disabled/loading
click-suppression, aria-busy/disabled, spinner, axe a11y) plus composeStories
fixtures. Config: define process.env.NODE_ENV for @testing-library/vue in the
browser, CI retry, drop the unusable @stories alias (tests use relative story
paths so validate-references resolves them). Removes the bootstrap smoke test.
…onents

Wave 1 — browser-mode (Chromium) functional coverage authored + self-verified
per component: actions (button-highlight, mini-button, copy-button,
segmented-button, icon-button), content (badge, card-box, card-pricing,
currency, avatar, tag, overline), feedback (message, empty-state,
status-indicator, skeleton, progress-bar, empty-results-block), inputs
(label, helper-text, chip), layout (divider), navigation (link), utils
(spinner). Each asserts rendering, emitted events with payloads,
disabled/loading suppression, ARIA, and axe a11y; composeStories where a
story exists. 419 tests green across 25 files. gitignore browser screenshots.
…ple components

Wave 2 — browser-mode functional coverage: inputs (input-text, input-number,
input-password, textarea, field-text, field-password, field-textarea,
checkbox, radio-button, switch, field-checkbox(-block), field-radio(-block),
field-switch(-block), box-grid-selection), data (code-block), navigation
(menu-item, breadcrumb-item), layout (sidebar + group/header/footer). Each
covers v-model round-trip, emitted events, disabled/readonly suppression,
ARIA, and axe; composeStories where a story exists.
…ay & recursive components

Wave 3 — browser-mode functional coverage for the hard components:
composition (item, table, paginator, flow, pick-list, log-view, toast,
select, multi-select, dropdown, tab-view, global-header, breadcrumb) and
overlays/recursive (dialog, drawer, panel, tooltip, navigation-menu) plus
calendar. Covers compound dot-notation, provide/inject shared state,
emitted events, v-model, open/close + Escape + focus-trap + Teleport for
overlays, nested-instance context for recursive menus, and axe a11y.
One breadcrumb overflow-select test is skipped with a documented reason.
Adds DOM globals (SVGElement, requestAnimationFrame, HTMLOptionElement) to
eslint config for the test files.
…apper (post-#715)

After dev's #715 merge, CopyButton wraps IconButton in a <span> carrying the
testid, so the testid element is the wrapper, not the <button>. Assert the
wrapper span contains the button instead of equaling it.
@robsongajunior robsongajunior requested a review from a team July 7, 2026 18:45
@robsongajunior robsongajunior requested a review from a team as a code owner July 7, 2026 18:45
@robsongajunior robsongajunior added do-not-merge WIP Work in Progress labels Jul 7, 2026
@robsongajunior robsongajunior changed the title feat: webkit component test [NO-ISSUE] fix(webkit): three-layer test architecture and the defects it surfaced Jul 7, 2026
@robsongajunior robsongajunior marked this pull request as draft July 7, 2026 19:15
Comment thread packages/webkit/src/components/code/log-view/log-view.test.ts Fixed
Comment thread packages/webkit/src/components/code/log-view/log-view.test.ts Fixed
Comment thread packages/webkit/src/components/code/log-view/log-view.test.ts Fixed
Comment thread packages/webkit/src/components/code/log-view/log-view.test.ts Fixed
Comment thread packages/webkit/src/components/content/accordion/accordion.test.ts Fixed
Comment thread packages/webkit/src/components/content/accordion/accordion.test.ts Fixed
Comment thread packages/webkit/src/components/content/accordion/accordion.test.ts Fixed
Comment thread packages/webkit/src/components/data/flow/flow.test.ts Fixed
Comment thread packages/webkit/src/components/data/flow/flow.test.ts Fixed
Comment thread packages/webkit/src/components/data/flow/flow.test.ts Fixed
@robsongajunior robsongajunior marked this pull request as ready for review July 10, 2026 17:46
robsongajunior and others added 3 commits July 10, 2026 16:48
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…defined'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…defined'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
@robsongajunior robsongajunior added WIP Work in Progress and removed WIP Work in Progress labels Jul 10, 2026
robsongajunior and others added 2 commits July 10, 2026 16:56
…defined'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…defined'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…defined'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Comment thread packages/webkit/src/components/navigation/breadcrumb/breadcrumb.test.ts Dismissed
Comment thread packages/webkit/src/components/navigation/breadcrumb/breadcrumb.test.ts Dismissed
Comment thread packages/webkit/src/components/navigation/breadcrumb/breadcrumb.test.ts Dismissed
Comment thread packages/webkit/src/components/navigation/breadcrumb/breadcrumb.test.ts Dismissed
Comment thread packages/webkit/src/components/navigation/breadcrumb/breadcrumb.test.ts Dismissed
Comment thread packages/webkit/src/components/navigation/breadcrumb/breadcrumb.test.ts Dismissed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WIP Work in Progress

Development

Successfully merging this pull request may close these issues.

3 participants