Replace hardcoded nextest binary names with a stable grouping mechanism

[coderabbitai]

Background

The nextest default-filter in .config/nextest.toml currently hardcodes the binary names behaviour_toolchain and behaviour_cli to exclude slow installer integration tests from local runs:

[profile.default]
default-filter = "not (binary(behaviour_toolchain) | binary(behaviour_cli))"

If these binaries are renamed or split in future, they will silently fall out of the filter and start running in the default profile again.

Problem

Hardcoded binary names are brittle. A rename or split of these test binaries would not cause any build or test failure — the filter would simply stop excluding them, meaning slow installer integration tests would start running in the default local profile without any warning.

Suggested Approaches

Consider introducing a more stable grouping mechanism, such as:

• A nextest test-group with a naming convention (e.g., all slow installer test binaries match binary(/^installer_/))
• A shared tag or package-level annotation that can be maintained in one place
• A comment in the test source files referencing the nextest config so renames trigger a review

Affected Files

• .config/nextest.toml — contains the hardcoded filter
• AGENTS.md — documents the profiles; will need updating if the mechanism changes
• docs/developers-guide.md — documents the profiles for contributors; will need updating if the mechanism changes

References

Raised as an overall comment in PR #179 (ci-only-installer-tests branch): #179

Raised by @leynos.

[leynos]

@CodeRabbit plan

[coderabbitai]

Coding Plan

Summary

• Rename the two slow installer test binaries to use a slow_ prefix (slow_toolchain.rs, slow_cli.rs)
• Replace the hardcoded nextest filter with a single regex pattern not binary(/^slow_/)
• Add a safeguard test that validates all slow_*.rs files are correctly excluded, catching accidental renames
• Update AGENTS.md and docs/developers-guide.md to document the naming convention

Design Choices

Design Choice 1: Which naming convention to use for slow test binaries?

Options Considered:

1. Prefix convention (slow_): Rename files to slow_toolchain.rs and slow_cli.rs; filter with binary(/^slow_/).
2. Suffix convention (_slow): Rename files to behaviour_toolchain_slow.rs and behaviour_cli_slow.rs; filter with binary(/_slow$/).
3. Subdirectory with explicit targets: Move slow tests to installer/tests/slow/ and define [[test]] targets with a common prefix.

Chosen Option: 1

Rationale: The slow_ prefix is immediately visible in file listings, unambiguous in its intent, and allows a single regex to capture all slow tests without complex escaping.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Rename Slow Installer Test Binaries

Adopt a slow_ prefix naming convention for all installer integration tests that should be excluded from the default nextest profile. This makes the filter self-documenting and allows future slow tests to be added simply by following the convention.

Task 1: Rename the slow test source files

Rename the two slow test binaries to follow the slow_ prefix convention.

• Rename installer/tests/behaviour_toolchain.rs → installer/tests/slow_toolchain.rs
• Rename installer/tests/behaviour_cli.rs → installer/tests/slow_cli.rs
• Rename the supporting directory installer/tests/behaviour_cli/ → installer/tests/slow_cli/
• Update the #[path] attribute in the renamed slow_cli.rs to reference slow_cli/scenarios.rs and slow_cli/support.rs

Task 2: Update internal documentation references

Fix doc comments that mention the old binary names.

• Update the doc comment in installer/tests/toolchain_steps/mod.rs to reference slow_toolchain.rs instead of behaviour_toolchain.rs

🤖 Prompt for AI agents

Rename the slow installer test binaries and their supporting files to adopt the
`slow_` prefix convention, and update all internal references accordingly.

- Rename `installer/tests/behaviour_toolchain.rs` to
`installer/tests/slow_toolchain.rs`
- Rename `installer/tests/behaviour_cli.rs` to `installer/tests/slow_cli.rs`
- Rename the supporting directory `installer/tests/behaviour_cli/` to
`installer/tests/slow_cli/`
- In the renamed `slow_cli.rs`, update the `#[path]` attribute to reference
`slow_cli/scenarios.rs` and `slow_cli/support.rs` (replacing any references to
the old `behaviour_cli/` paths)
- In `installer/tests/toolchain_steps/mod.rs`, update the doc comment to
reference `slow_toolchain.rs` instead of `behaviour_toolchain.rs`

Phase 2: Update Nextest Configuration

Migrate all nextest filters and overrides from hardcoded binary names to regex patterns matching the slow_ prefix.

Task 1: Update the default-filter to use regex

Replace the hardcoded binary names with a single regex pattern.

• In .config/nextest.toml, change the default profile filter from:
  not (binary(behaviour_toolchain) | binary(behaviour_cli))
  to:
  not binary(/^slow_/)

Task 2: Update overrides referencing the old binary names

The nextest config contains overrides that reference behaviour_toolchain by name for slow-timeout and the serial test-group.

• Update the slow-timeout override filter from binary(behaviour_toolchain) to binary(slow_toolchain)
• Update the serial-toolchain-installs test-group filter from binary(behaviour_toolchain) & (test(...)) to binary(slow_toolchain) & (test(...))

🤖 Prompt for AI agents

Update `.config/nextest.toml` to replace all hardcoded binary name references
with regex-based or updated literal patterns matching the `slow_` prefix
convention.

- In the `default` profile, replace the `default-filter` value:
  - Old: `not (binary(behaviour_toolchain) | binary(behaviour_cli))`
  - New: `not binary(/^slow_/)`
- Update the `slow-timeout` override filter:
  - Old: `binary(behaviour_toolchain)`
  - New: `binary(slow_toolchain)`
- Update the `serial-toolchain-installs` test-group filter:
  - Old: `binary(behaviour_toolchain) & (test(...))`
  - New: `binary(slow_toolchain) & (test(...))`

Do not alter any other settings or profiles in the file.

Phase 3: Add Safeguard Test and Update Documentation

Add automated validation to catch future renames that break the convention, and update all documentation to reflect the new mechanism.

Task 1: Add safeguard test for slow test discovery

Create a test that validates all slow_*.rs files in installer/tests/ are correctly matched by the nextest default-filter.

• Create a new test in tests/nextest_slow_filter.rs (or extend tests/nextest_ui_filter.rs)
• The test should:
  • Enumerate all files matching installer/tests/slow_*.rs
  • Parse .config/nextest.toml and extract the default profile's default-filter
  • Assert the filter uses a regex pattern that would exclude all discovered slow binaries
  • Fail with a descriptive message if any slow test binary would not be excluded
• Follow the existing pattern from nextest_ui_filter.rs for parsing the TOML configuration

Task 2: Update AGENTS.md

Reflect the new naming convention in the agents documentation.

• Update the section describing slow installer test exclusion to reference the slow_ prefix convention
• Replace references to behaviour_toolchain and behaviour_cli with slow_toolchain and slow_cli
• Add a note explaining that any new slow test must use the slow_ prefix to be excluded from the default profile

Task 3: Update docs/developers-guide.md

Update the contributor documentation with the new mechanism.

• Update the "Test profiles" subsection to describe the regex-based filter
• Remove the warning about hardcoded binary names requiring manual filter updates
• Document the slow_ prefix convention and explain that new slow tests automatically inherit the exclusion by following the naming pattern
• Update the table describing what each profile runs to reflect the new filter syntax

🤖 Prompt for AI agents

Implement a safeguard test that validates the `slow_` naming convention is
correctly captured by the nextest default-filter, then update all relevant
documentation.

**Safeguard test (`tests/nextest_slow_filter.rs` or extend
`tests/nextest_ui_filter.rs`)**:
- Follow the existing pattern from `tests/nextest_ui_filter.rs` for parsing the
TOML configuration
- Enumerate all files matching `installer/tests/slow_*.rs`
- Parse `.config/nextest.toml` and extract the `default` profile's
`default-filter` value
- Assert that the filter uses a regex pattern that would exclude all discovered
slow binaries
- Fail with a descriptive message if any slow test binary would not be excluded

**`AGENTS.md`**:
- Update the section describing slow installer test exclusion to reference the
`slow_` prefix convention
- Replace all references to `behaviour_toolchain` and `behaviour_cli` with
`slow_toolchain` and `slow_cli`
- Add a note stating that any new slow test must use the `slow_` prefix to be
excluded from the default profile

**`docs/developers-guide.md`**:
- Update the "Test profiles" subsection to describe the regex-based filter
- Remove the warning about hardcoded binary names requiring manual filter
updates
- Document the `slow_` prefix convention, explaining that new slow tests
automatically inherit the exclusion by following the naming pattern
- Update the table describing what each profile runs to reflect the new filter
syntax

Research

The workspace uses nextest for test orchestration, configured via .config/nextest.toml. Two profiles exist: default (excludes slow installer tests) and ci (runs everything). The installer integration tests live in installer/tests/ and are auto-discovered by Cargo—no explicit [[test]] sections exist. A precedent for validating nextest configuration exists in tests/nextest_ui_filter.rs, which asserts the serial-dylint-ui filter captures the correct binaries.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Rename the slow installer test binaries and their supporting files to adopt the
`slow_` prefix convention, and update all internal references accordingly.

- Rename `installer/tests/behaviour_toolchain.rs` to
`installer/tests/slow_toolchain.rs`
- Rename `installer/tests/behaviour_cli.rs` to `installer/tests/slow_cli.rs`
- Rename the supporting directory `installer/tests/behaviour_cli/` to
`installer/tests/slow_cli/`
- In the renamed `slow_cli.rs`, update the `#[path]` attribute to reference
`slow_cli/scenarios.rs` and `slow_cli/support.rs` (replacing any references to
the old `behaviour_cli/` paths)
- In `installer/tests/toolchain_steps/mod.rs`, update the doc comment to
reference `slow_toolchain.rs` instead of `behaviour_toolchain.rs`
===============================================================================

Task: 2

Update `.config/nextest.toml` to replace all hardcoded binary name references
with regex-based or updated literal patterns matching the `slow_` prefix
convention.

- In the `default` profile, replace the `default-filter` value:
  - Old: `not (binary(behaviour_toolchain) | binary(behaviour_cli))`
  - New: `not binary(/^slow_/)`
- Update the `slow-timeout` override filter:
  - Old: `binary(behaviour_toolchain)`
  - New: `binary(slow_toolchain)`
- Update the `serial-toolchain-installs` test-group filter:
  - Old: `binary(behaviour_toolchain) & (test(...))`
  - New: `binary(slow_toolchain) & (test(...))`

Do not alter any other settings or profiles in the file.
===============================================================================

Task: 3

Implement a safeguard test that validates the `slow_` naming convention is
correctly captured by the nextest default-filter, then update all relevant
documentation.

**Safeguard test (`tests/nextest_slow_filter.rs` or extend
`tests/nextest_ui_filter.rs`)**:
- Follow the existing pattern from `tests/nextest_ui_filter.rs` for parsing the
TOML configuration
- Enumerate all files matching `installer/tests/slow_*.rs`
- Parse `.config/nextest.toml` and extract the `default` profile's
`default-filter` value
- Assert that the filter uses a regex pattern that would exclude all discovered
slow binaries
- Fail with a descriptive message if any slow test binary would not be excluded

**`AGENTS.md`**:
- Update the section describing slow installer test exclusion to reference the
`slow_` prefix convention
- Replace all references to `behaviour_toolchain` and `behaviour_cli` with
`slow_toolchain` and `slow_cli`
- Add a note stating that any new slow test must use the `slow_` prefix to be
excluded from the default profile

**`docs/developers-guide.md`**:
- Update the "Test profiles" subsection to describe the regex-based filter
- Remove the warning about hardcoded binary names requiring manual filter
updates
- Document the `slow_` prefix convention, explaining that new slow tests
automatically inherit the exclusion by following the naming pattern
- Update the table describing what each profile runs to reflect the new filter
syntax

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!