Add automated end-to-end testing using playground in CI/CD workflows

[mfogliatto]

Overview

Extend the playground project to support automated end-to-end testing that validates ReferenceCop's key scenarios in PR and official build workflows.

Context

Currently, the playground project in playground/ provides a manual testing environment where developers can:

• Build the ReferenceCop package
• Test it against sample projects with various rule violations
• Validate changes end-to-end before committing

This works well for manual validation, but we should automate this to ensure all key scenarios are tested on every check-in.

Goals

1. Create automated test scenarios that validate all ReferenceCop rule types:

   • AssemblyName rules (e.g., blocking Newtonsoft.Json)
   • ProjectTag rules (e.g., preventing App projects from referencing Tools)
   • ProjectPath rules (e.g., preventing App from referencing Internal libraries)

2. Integrate these tests into GitHub Actions workflows:

   • Run on every PR build
   • Run on official builds
   • Fail the build if any scenario doesn't produce expected results

3. Test both positive and negative scenarios:

   • Verify violations are correctly detected (build fails with expected diagnostics)
   • Verify valid references are allowed (build succeeds)

Implementation Approach

Option 1: PowerShell Test Runner

Create a test runner script that:

• Builds various test configurations in the playground
• Validates expected build outcomes (success/failure)
• Checks for expected diagnostic messages
• Returns appropriate exit codes for CI/CD

Option 2: Integration Test Project

Create a proper MSTest/xUnit test project that:

• Uses the playground projects as test fixtures
• Invokes MSBuild programmatically
• Asserts on build results and diagnostics
• Integrates with existing test infrastructure

Option 3: Combination

• Use PowerShell for simple scenario validation
• Add integration tests for complex validation logic
• Both run in CI/CD pipeline

Acceptance Criteria

• Automated tests validate all three rule types (AssemblyName, ProjectTag, ProjectPath)
• Tests verify both violations (build failure) and valid scenarios (build success)
• Tests check for expected diagnostic codes (RC0001, RC0002) and messages
• Tests run in PR build workflow (.github/workflows/pr-build.yaml or similar)
• Tests run in official build workflow (.github/workflows/official-build.yaml)
• Failed tests cause the CI/CD build to fail
• Test results are visible in GitHub Actions UI
• Documentation updated with automated testing approach

Benefits

• Catch regressions early in the development cycle
• Ensure all rule types work correctly across releases
• Provide confidence when accepting community contributions
• Validate package integration scenarios automatically
• Reduce manual testing effort

Related

• Playground: playground/
• Test scenarios reference: playground/TEST-SCENARIOS.md
• Build workflows: .github/workflows/