feat: Playwright E2E UI test suite for Magma views#3283
Closed
deacon-mp wants to merge 2 commits into
Closed
Conversation
Adds comprehensive browser-level tests in tests/e2e/magma/ that verify
the Magma Vue UI matches what the backend API returns.
Infrastructure (conftest.py):
- Finds a free port automatically (via socket.bind port=0) so multiple
VENVs/CI jobs can run in parallel without port conflicts
- Writes conf/local.yml with the chosen port BEFORE starting the server
(Caldera loads config into memory at startup; the file must be written
while the server is stopped and read on the next start)
- Backs up and restores any pre-existing conf/local.yml
- Refuses to start if the chosen port is already in use
- Provides auth_page fixture with session cookies pre-loaded
Test coverage (14 view files, ~100 tests total):
- test_login.py — login form, valid/invalid creds, redirect behaviour
- test_home.py — dashboard boxes, navigation links, config API
- test_agents.py — agent table, deploy/config/bulk-action buttons
- test_abilities.py — search/filter controls, create modal, API data match
- test_adversaries.py — selector dropdown, new profile, search filter
- test_operations.py — create/delete/download buttons, selector dropdown
- test_fact_sources.py — source list, CRUD buttons, fact/rule tables
- test_payloads.py — payload list, upload input, API upload round-trip
- test_schedules.py — create modal, schedule list, cron expression display
- test_objectives.py — selector, new objective, detail panel
- test_planners.py — read-only list, API data match
- test_contacts.py — read-only list, API data match
- test_obfuscators.py — plain-text and base64 always present
- test_settings.py — port in page, code editor, API key not exposed
New files:
- requirements-ui-tests.txt (playwright, pytest-playwright, nodeenv)
- tox.ini [testenv:ui] (nodeenv → npm build → playwright → pytest)
Run with: tox -e ui
pytest tests/e2e/magma -v --browser chromium
E2E Playwright tests require playwright and browser binaries not available in the standard test environment. They are run via tox -e ui instead.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Playwright-based end-to-end UI test suite for the Magma (Vue) frontend, plus a dedicated tox environment to build the frontend and run those tests against a locally started Caldera instance.
Changes:
- Introduces a new
tox -e uienvironment to provision Node.js, build Magma assets, install Playwright browsers, and execute E2E tests. - Adds a Playwright/pytest E2E test suite covering multiple Magma routes (login, home, agents, operations, etc.).
- Adds a separate requirements file for UI test dependencies.
Reviewed changes
Copilot reviewed 17 out of 19 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tox.ini | Adds a ui tox environment to build Magma + run Playwright E2E tests |
| requirements-ui-tests.txt | Defines Python dependencies for Playwright E2E testing |
| tests/e2e/init.py | Marks tests.e2e as a package |
| tests/e2e/magma/init.py | Marks tests.e2e.magma as a package |
| tests/e2e/magma/conftest.py | Adds fixtures to start Caldera, create an authenticated API session, and provide an authenticated Playwright page |
| tests/e2e/magma/test_login.py | E2E tests for login flow and redirects |
| tests/e2e/magma/test_home.py | E2E tests for authenticated home/dashboard behavior |
| tests/e2e/magma/test_agents.py | E2E tests for agents view structure/interactions |
| tests/e2e/magma/test_operations.py | E2E tests for operations view and dropdown behavior |
| tests/e2e/magma/test_adversaries.py | E2E tests for adversaries view, including create/delete cleanup |
| tests/e2e/magma/test_abilities.py | E2E tests for abilities view, filters, and modal behavior |
| tests/e2e/magma/test_objectives.py | E2E tests for objectives view, including create/delete cleanup |
| tests/e2e/magma/test_fact_sources.py | E2E tests for fact sources view, including create/delete cleanup |
| tests/e2e/magma/test_payloads.py | E2E tests for payload listing and upload lifecycle cleanup |
| tests/e2e/magma/test_planners.py | E2E tests for planners view (read-only expectations) |
| tests/e2e/magma/test_schedules.py | E2E tests for schedules view and API/UI parity checks |
| tests/e2e/magma/test_settings.py | E2E tests for settings view, config display, and masking checks |
| tests/e2e/magma/test_contacts.py | E2E tests for contacts view (read-only expectations) |
| tests/e2e/magma/test_obfuscators.py | E2E tests for obfuscators view (read-only expectations) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # 3. Install Playwright browser binaries (Chromium only for speed) | ||
| playwright install chromium | ||
| # 4. Run the E2E test suite | ||
| pytest plugins/magma/tests/e2e --tb=short -v {posargs} |
Comment on lines
+40
to
+45
| _HERE = os.path.dirname(__file__) | ||
| CALDERA_ROOT = os.path.normpath(os.path.join(_HERE, '..', '..', '..', '..', '..')) | ||
| CONF_DIR = os.path.join(CALDERA_ROOT, 'conf') | ||
| DEFAULT_YML = os.path.join(CONF_DIR, 'default.yml') | ||
| LOCAL_YML = os.path.join(CONF_DIR, 'local.yml') | ||
|
|
| import shutil | ||
| import socket | ||
| import subprocess | ||
| import tempfile |
Comment on lines
+80
to
+92
| If conf/local.yml already exists it is backed up first so the original | ||
| is restored in teardown. | ||
| """ | ||
| with open(DEFAULT_YML, 'r', encoding='utf-8') as fh: | ||
| config = yaml.safe_load(fh) | ||
|
|
||
| config['port'] = port | ||
| config['host'] = '127.0.0.1' | ||
|
|
||
| with open(LOCAL_YML, 'w', encoding='utf-8') as fh: | ||
| yaml.safe_dump(config, fh, default_flow_style=False) | ||
|
|
||
|
|
Comment on lines
+229
to
+237
| pw_cookies = [ | ||
| { | ||
| 'name': c.name, | ||
| 'value': c.value, | ||
| 'domain': '127.0.0.1', | ||
| 'path': '/', | ||
| 'httpOnly': False, | ||
| 'secure': False, | ||
| } |
| pytest plugins/magma/tests/e2e/test_home.py -v --browser chromium | ||
| """ | ||
|
|
||
| import pytest |
|
|
||
| if api_count == 0: | ||
| # Neither selector should produce any visible results | ||
| assert tbody_rows.count() == 0 or list_items.count() == 0, ( |
Comment on lines
+183
to
+195
| # Identify sources that did not exist before (by comparing API count) | ||
| # Fetch the pre-test API state to compare IDs would require an earlier | ||
| # API call; instead we delete the source whose name is the default | ||
| # auto-generated name (typically "New Source" or similar). If multiple | ||
| # unnamed sources exist, we delete only the most recently created one. | ||
| if len(sources_after) > initial_count: | ||
| # Sort by id descending to get the newest entry first | ||
| newest = sorted(sources_after, key=lambda s: s.get('id', ''), reverse=True)[0] | ||
| del_resp = api_session.delete(base_url + f'/api/v2/sources/{newest["id"]}') | ||
| assert del_resp.status_code in (200, 204), ( | ||
| f'Cleanup DELETE /api/v2/sources/{newest["id"]} returned ' | ||
| f'HTTP {del_resp.status_code}' | ||
| ) |
Contributor
Author
|
Closing this PR as the Playwright E2E tests for the Magma frontend belong in the Magma plugin repository (mitre/magma), not core Caldera. The tests have been moved to mitre/magma#92. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/e2e/magma/with 14 Playwright test files (~100 tests) covering every Magma UI viewtox -e uienvironment that installs Node vianodeenv, builds Magma (npm run build), installs Playwright Chromium, then runs the suiterequirements-ui-tests.txtforplaywright,pytest-playwright,nodeenvKey design decisions
Port isolation (multiple VENVs / CI jobs run in parallel):
conftest.pyfinds a free port viasocket.bind(port=0), writesconf/local.ymlwith that port before starting the server, and cleans up after. Since Caldera loads all config into memory at startup,local.ymlis only written/deleted while the server is fully stopped — never during a running session.Structure: Tests live in
tests/e2e/magma/(mirrors plugin organisation;plugins/magmais a submodule so files must be in the main repo).Test coverage
Test plan
tox -e uipasses on a fresh VENV with Node 20 available🤖 Generated with Claude Code