Add Appium and Avalonia headless UI test tiers with cross-platform support#17
Merged
Merged
Conversation
…tier - Add test/DemaConsulting.SysML2Workbench.UiTests: Avalonia.Headless + xUnit headless UI tests exercising view/viewmodel behavior without a real window, runnable on all OSes with no special setup. - Add test/DemaConsulting.SysML2Workbench.IntegrationTests: Appium.WebDriver based end-to-end tests driving the compiled Desktop app through the OS accessibility tree, following the Avalonia Appium testing docs. AppFixture branches by OS: Windows via NovaWindows driver (implemented and exercised), macOS (Mac2) and Linux (AT-SPI) paths are structurally present but intentionally unimplemented/skipped pending future CI provisioning. - Add AutomationProperties.AutomationId to key interactive controls (menu items, dock control, About dialog OK button, workspace panel tree/buttons, predefined views list) so Appium tests can reliably locate elements. - Update .github/workflows/build.yaml: filter existing cross-platform test job(s) to Category!=Integration, add new windows-latest job that builds/publishes the Desktop app, installs Appium + appium-novawindows-driver, runs the Appium server, and runs the IntegrationTests project. Apply the same filter in build.ps1. - Add both new projects to SysML2Workbench.slnx. - Update docs/design/introduction.md and add docs/design/ots/appium.md, docs/verification/ots/appium.md, docs/reqstream/ots/appium.yaml, docs/sysml2/model/ots.sysml entry, requirements.yaml include, .cspell.yaml terms, and README.md to describe the new three/four-tier test strategy and its Windows-only CI automation scope. - Update .reviewmark.yaml with a new OTS-Appium review-set and extend the MainWindowShell/WorkspacePanel review-sets to cover the new UiTests files.
…option
The Windows session builder was calling AddAdditionalAppiumOption("app", ...)
which throws at runtime (ArgumentException: 'There is already an option for
the appium:app capability. Please use the Application property instead.')
because Appium.WebDriver already reserves that capability name. Discovered by
actually running an integration test against a local Appium + NovaWindows
driver session. Fixed by setting the built-in AppiumOptions.App property
instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on-test steps - build.ps1 now accepts -Build, -Test, -IntegrationTest, -All switches instead of always running the same restore+build+test sequence - No args prints usage and exits non-zero - -Test and -IntegrationTest are self-sufficient (build the solution themselves if not already done in the same invocation) - -IntegrationTest mirrors the CI appium-windows-integration-tests job: installs Appium/NovaWindows driver, publishes the Desktop app, starts an Appium server (direct node invocation for reliable PID tracking/cleanup), runs the IntegrationTests project, and always stops the Appium server via try/finally - Fix build.yaml's Appium server start step to use appium.cmd and the correct --allow-insecure '*:chromedriver_autodownload' CLI syntax - Update docs (design/introduction.md, design/ots/appium.md, README.md, toolchain_setup.md, developer.agent.md, AGENTS.md) to describe the new build.ps1 switch-driven usage
AppFixture.cs no longer starts or manages an Appium/AT-SPI server itself on any OS - it always just connects a WebDriver client to http://127.0.0.1:4723, trusting that something already made a server available there. A new run-under-appium.ps1 owns that server's lifecycle uniformly: - Windows/macOS: starts a local Appium server (resolving node/appium directly), polls until ready, runs the wrapped command, then always stops the server. - Linux: delegates the whole wrapped command to KDE's selenium-webdriver-at-spi-run, since that backend has no standalone, directly-startable server binary - only its -run wrapper, which itself boots a nested Wayland session and runs the wrapped command as its child. build.ps1 -IntegrationTest is now cross-platform: it installs Appium plus the NovaWindows/Mac2 driver on Windows/macOS (skipping driver install on Linux, where selenium-webdriver-at-spi must already be built and installed), then publishes the Desktop app for the current OS/RID and runs ./run-under-appium.ps1 -- dotnet test ... Also fixes a driver-install argument-concatenation bug and a PowerShell exit-code-capture footgun (capturing a function's return value while it also runs a native command mixed stdout into the return array, silently turning test failures into a 0 exit code). .github/workflows/build.yaml's appium-windows-integration-tests job now delegates to the same run-under-appium.ps1 script instead of duplicating the start/poll/stop logic inline, while remaining Windows-only in CI. Added a minimal LinuxDriver : AppiumDriver subclass in AppFixture.cs since Appium.WebDriver 8.x's AppiumDriver base class has no platform-agnostic/generic concrete form and its (Uri, ICapabilities) constructor isn't accessible outside its own assembly. Removed SkippableOnNonWindowsFactAttribute and reverted MainWindowShellIntegrationTests to plain [Fact], since AppFixture now genuinely attempts all three OS paths instead of throwing on macOS/Linux. Verified end-to-end via build.ps1 -IntegrationTest on Windows: 3/3 IntegrationTests pass, no orphaned processes, plus a full build.ps1 -Test run (255 unit + 4 UiTests + 24 OTS tests, all passing). Updated docs/design/introduction.md, docs/design/ots/appium.md, and docs/verification/ots/appium.md to describe the new cross-platform design. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ran the built-in change-review agent against the feature/appium-integration-tests diff, then held formal-reviews (gpt-5.4-mini) of the review-sets affected by this branch's changes: Purpose, Decomposition, SysML2Workbench-Architecture, SysML2Workbench-Design, OTS-Appium, and the AppShellSubsystem MainWindowShell/ WorkspacePanel/AboutDialog unit sets. Confirmed via dotnet reviewmark --lint and dotnet reviewmark --plan --enforce that no .reviewmark.yaml updates were needed - all files changed by this branch are already covered by the OTS-Appium and AppShellSubsystem review-sets from an earlier commit. Fixes for findings directly caused by/tightly coupled to this branch's changes: - run-under-appium.ps1: fixed a PowerShell range-slice edge case ([1..(.Count - 1)] silently returned the single element instead of an empty array when only one command token was supplied), duplicating the wrapped command's own name as a spurious argument for zero-argument wrapped commands. - README.md: updated the stale IntegrationTests description (was Windows-only/NovaWindows-only) to describe the current cross-platform Appium/AT-SPI behavior via run-under-appium.ps1. - docs/design/ots/appium.md: removed a recorded Avalonia version number, which violates the repository's design-documentation standard. - MainWindowShellIntegrationTests.cs: renamed DesktopApp_FileMenu_AddFileSourceMenuItem_IsDiscoverableAndClickable to ...IsDiscoverableAndEnabled and added an Enabled assertion, since the test never actually clicks the menu item (clicking it would open an OS-native file dialog outside Avalonia's accessibility tree). - .cspell.yaml: added three legitimate technical terms introduced by this branch's doc/code changes (unvalidated, startable, automatable). Pre-existing findings from the same review pass (stale unit/subsystem counts in docs/design/introduction.md, incomplete OTS dependency lists, compound requirements, missing .NET 9.0 design-doc coverage, a stale SysML2Tools version number in layout-invoker.md, and non-absolute README links) predate this branch and are out of scope for this change; they are documented in the saved review reports under .agent-logs/reviews/ for separate follow-up. Ran the lint-fix agent last: pwsh ./fix.ps1 made no changes (already clean), and pwsh ./lint.ps1 passed with exit code 0 after the .cspell.yaml update. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The formal-review fixup commit renamed DesktopApp_FileMenu_AddFileSourceMenuItem_IsDiscoverableAndClickable to ...IsDiscoverableAndEnabled in MainWindowShellIntegrationTests.cs, but left docs/reqstream/ots/appium.yaml's Appium-LocateControlsByAutomationId requirement referencing the old test name, and docs/verification/ots/appium.md still describing the old name. This caused CI's reqstream --enforce step to report the requirement as untested (Only 111 of 112 requirements are satisfied with tests). Updated both to the current test name, and added a short verification note explaining why the test intentionally does not click the menu item. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a cross-platform, tiered build and test strategy, with major improvements to how integration tests are run and documented. The changes add first-class support for Appium-driven end-to-end integration tests (currently run on Windows in CI, but with local support for macOS and Linux), refactor the build scripts to allow selective execution of build and test tiers, and update documentation and configuration to reflect these improvements. Additionally, review coverage and spell-checking are updated to include the new integration test infrastructure.
Build and Test Pipeline Improvements
build.ps1to support combinable switches:-Build,-Test,-IntegrationTest, and-All, allowing selective execution of build, unit/headless tests, and Appium/AT-SPI integration tests. The script now installs Appium and the appropriate driver on Windows/macOS, publishes the Desktop app, and runs integration tests viarun-under-appium.ps1..github/workflows/build.yaml) to add a dedicatedappium-windows-integration-testsjob that builds and publishes the Desktop app, installs Appium and the NovaWindows driver, and runs the Appium-driven integration tests on Windows. The default test job now excludes integration tests with--filter "Category!=Integration"[1] [2] [3].Documentation Updates
README.md,AGENTS.md, anddocs/build_notes/toolchain_setup.mdto document the new build and test workflow, clarify the role of integration tests, and explain how to run each test tier locally and in CI [1] [2] [3] [4].Test and Review Coverage
UiTestsandIntegrationTeststoSysML2Workbench.slnx..reviewmark.yamlto include new test files and add a review item for Appium functionality and its associated test and documentation files [1] [2] [3].Spell Checking
.cspell.yamlto prevent false positives in spell checking.These changes collectively enable robust, platform-aware build and test workflows, with clear documentation and review coverage for the new integration test tier.