Code Coverage automatic enablement (Code Quality) [GA] (#61587)

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Melanie Yarbrough <11952755+myarb@users.noreply.github.com>
Co-authored-by: AlonaHlobina <54394529+AlonaHlobina@users.noreply.github.com>
Copilot-Session: cc516c8e-bdeb-4490-85c3-6ef91946419a

Author: 6 people

content/code-security/concepts/code-quality/automatic-code-coverage-setup.md

@@ -0,0 +1,48 @@
+---
+title: Automatic code coverage setup
+shortTitle: Automatic code coverage
+intro: 'An AI-powered agent can analyze your repository and generate a working code coverage workflow, so you can start tracking test coverage without manually authoring CI configuration.'
+versions:
+  feature: code-quality
+product: '{% data reusables.gated-features.code-quality-availability %}'
+contentType: concepts
+category:
+  - Improve code quality
+---
+
+When you use automatic setup for code coverage, an AI-powered agent analyzes your repository, identifies your test framework, and opens a pull request with a coverage workflow ready for review. 
+
+**There is no additional cost for using this feature.**
+
+## How the agent works
+
+The agent works in three phases:
+
+1. **Discovery:** The agent reads your CI configuration, documentation, and build files to understand your project structure and identify your test framework.
+1. **Execution:** The agent installs dependencies, builds the project, and runs your tests with coverage enabled. If coverage tooling is not already configured, the agent adds it to your project configuration (for example, `vitest.config.ts` or `jest.config.js`).
+1. **Workflow integration:** If the agent produces a valid coverage report, it checks whether your repository already has a {% data variables.product.prodname_actions %} workflow that runs tests on pull requests. If so, the agent augments that workflow with a coverage upload step. If not, it creates a new workflow file and opens a pull request.
+
+## When the agent stops
+
+The agent may stop before opening a pull request in the following situations:
+
+* **No tests found.** The agent couldn't find tests to instrument, so there's nothing to generate coverage for.
+* **Can't reproduce the build.** Missing private registries, proprietary SDKs, or system dependencies prevent the agent from verifying the test suite.
+
+If the agent stops or produces unexpected results, you can review the agent's session log for details. Navigate to the **Tasks** tab in your repository to find the session associated with the workflow generation attempt.
+* **Unsupported coverage report conversion.** The agent won't reconstruct Cobertura XML from reports that only expose aggregated counters. For example, JaCoCo XML does not contain enough line and branch structure for a trustworthy Cobertura upload, so JVM projects that only produce JaCoCo XML may need manual setup instead.
+## Pull request outcomes
+
+> [!NOTE]
+> The agent opens the pull request immediately with an initial planning commit that contains no code changes. The actual implementation commit typically arrives a few minutes later. If the pull request initially shows 0 changed files, wait a few minutes and refresh the page.
+
+If the agent successfully opens a pull request, the pull request may be in one of these states:
+
+* **Mergeable as-is:** The workflow completes successfully in CI and coverage uploads correctly.
+* **Ready to iterate:** The workflow runs but requires adjustments (for example, missing secrets, self-hosted runner configuration, or path differences between local verification and CI).
+* **Useful as a reference:** Maintainers may prefer to configure coverage themselves, using the agent's pull request as a starting point for the build and test commands it discovered.
+
+## Further reading
+
+* [AUTOTITLE](/code-security/how-tos/maintain-quality-code/set-up-code-coverage#automatic-setup)
+* [AUTOTITLE](/code-security/concepts/code-quality/code-quality)

content/code-security/concepts/code-quality/index.md

@@ -8,4 +8,5 @@ contentType: concepts
 children:
   - /code-quality
   - /enablement-at-scale
+  - /automatic-code-coverage-setup
 ---

content/code-security/how-tos/maintain-quality-code/enable-code-quality.md

@@ -59,5 +59,5 @@ If you're rolling out the feature across many teams, we recommend you pilot on a
 
 ## Next steps
 
-* **Add code coverage:** Upload reported code coverage from your test suite to see coverage results directly on pull requests. See [AUTOTITLE](/code-security/how-tos/maintain-quality-code/set-up-code-coverage).
+* **Add code coverage:** Upload reported code coverage from your test suite to see coverage results directly on pull requests. {% data reusables.code-quality.workflow-generation %}
 * **For your organization:** Understand the code health of your repositories at a glance. See [AUTOTITLE](/code-security/how-tos/maintain-quality-code/explore-code-quality).

content/code-security/how-tos/maintain-quality-code/set-up-code-coverage.md

@@ -14,15 +14,55 @@ category:
   - Improve code quality
 ---
 
+You can set up code coverage for your repository in two ways: 
+
+* **Automatic setup:** Use the AI-powered agent to generate a workflow automatically. Choose this option if:
+  * You want to get started quickly without writing YAML configuration.
+  * Your project uses common test frameworks and build patterns.
+  * You're comfortable iterating on an AI-generated workflow.
+* **Manual setup:** Configure your CI workflow yourself. Choose this option if:
+  * You need precise control over the coverage process.
+  * You have complex CI requirements (such as private registries or custom build steps).
+  * You want to understand exactly how coverage is configured.
+
+## Automatic setup
+
+You can use the automatic setup option to generate a working code coverage workflow without manually authoring CI configuration. An agent analyzes your repository, identifies your test framework, and opens a pull request with a coverage workflow ready for review.
+
+> [!NOTE]
+> Automatic setup uses AI to generate the workflow file. There is no additional cost for using this feature.
+
+### Prerequisites for automatic setup
+
+* {% data variables.product.prodname_code_quality_short %} is enabled for your repository. See [AUTOTITLE](/code-security/how-tos/maintain-quality-code/enable-code-quality).
+* Your repository has an existing test suite.
+
+### Generating a coverage workflow automatically
+
+{% data reusables.repositories.navigate-to-repo %}
+{% data reusables.repositories.sidebar-settings %}
+1. In the sidebar, under "Security", click **{% data variables.code-quality.code_quality_ui_settings %}** to display the "{% data variables.code-quality.code_quality_ui %}" page.
+1. In the "Code coverage analysis" section, click the **Setup** dropdown box.
+1. In the list, select **Generate workflow with AI**. Wait for the agent to analyze your repository. The agent opens a draft pull request and posts a checklist of the steps it is working through.
+1. To review the pull request, click **Review pull request**. 
+   Review the pull request once the agent completes its work. The pull request description summarizes the changes, including any project configuration updates, workflow file changes, and coverage output settings.
+1. If the workflow runs successfully in CI and coverage uploads correctly, merge the pull request.
+
+   If the workflow needs adjustments, see [AUTOTITLE](/code-security/concepts/code-quality/automatic-code-coverage-setup) for guidance on the different outcomes and how to iterate.
+
+For more information about how the agent works and what to expect, see [AUTOTITLE](/code-security/concepts/code-quality/automatic-code-coverage-setup).
+
+## Manual setup
+
 Built-in code coverage lets you track how thoroughly your tests exercise your code, without adding a third-party service to your toolchain or budget. In the following procedures, you will generate a Cobertura XML coverage report from your test suite, upload it to {% data variables.product.github %}, and view the coverage results on your pull requests.
 
-## Prerequisites
+### Prerequisites for manual setup
 
 * {% data variables.product.prodname_code_quality_short %} is enabled for your repository.
 * Your repository has a test suite that runs in {% data variables.product.prodname_actions %}.
 * Your test framework can produce a coverage report in **Cobertura XML** format.
 
-## Step 1: Generate a Cobertura XML coverage report
+### Step 1: Generate a Cobertura XML coverage report
 
 Configure your test framework to output a coverage report in the Cobertura XML format. Code coverage works with any programming language that can produce this format.
 
@@ -40,7 +80,7 @@ Configure your test framework to output a coverage report in the Cobertura XML f
 > [!TIP]
 > If your framework isn't listed above, check its documentation for Cobertura output support. Many tools either support it directly or can convert to Cobertura XML from other formats.
 
-## Step 2: Upload the coverage report
+### Step 2: Upload the coverage report
 
 After your tests generate a Cobertura XML report, upload it to {% data variables.product.github %} so coverage results appear on pull requests.
 
@@ -117,7 +157,7 @@ jobs:
           label: code-coverage/pytest
 ```
 
-## Step 3: View coverage results on pull requests
+### Step 3: View coverage results on pull requests
 
 1. Open a pull request (or push to an existing one) that triggers the workflow you configured.
 1. After the workflow completes, look for a comment from `{% data variables.code-quality.pr_commenter %}` on the pull request. The comment includes:

content/code-security/responsible-use/security-and-quality-ai-features.md

@@ -40,7 +40,7 @@ GitHub's security and quality platform includes several AI-powered capabilities
 * **Copilot Autofix for code scanning**: Automatically generates fix suggestions for CodeQL alerts on pull requests and the default branch.
 * **Generic secret detection**: Uses a model to identify unstructured secrets in source code that deterministic pattern matching cannot find.
 * **Custom pattern regex generator**: Uses AI to generate regular expressions for custom secret scanning patterns from natural language descriptions.
-* **GitHub Code Quality**: Surfaces code quality issues and offers LLM-powered fix suggestions on pull requests and the default branch.
+* **GitHub Code Quality**: Surfaces code quality issues and offers LLM-powered fix suggestions on pull requests and the default branch. Also provides AI-powered automatic generation of code coverage workflows.
 
 Copilot Autofix is an expansion of code scanning that provides users with targeted recommendations to help them fix code scanning alerts, avoiding the introduction of new security vulnerabilities. Potential fixes are generated automatically by large language models (LLMs) using data from the codebase and from code scanning analysis. Copilot Autofix is available for CodeQL analysis and does not require a GitHub Copilot subscription.
 
@@ -50,7 +50,7 @@ Secret scanning's generic secret detection is an AI-powered expansion of secret
 
 Secret scanning's custom pattern regular expression generator makes it possible to define custom secret scanning patterns without knowledge of regular expressions. Users input a natural language description of what they want to detect, along with optional example strings, and the generator produces up to three candidate regular expressions. These patterns can then be validated via the dry-run mechanism before being deployed as custom patterns. The regular expression generator does not require a GitHub Copilot subscription.
 
-GitHub Code Quality helps users improve code reliability, maintainability, and overall project health by surfacing actionable feedback and offering automatic fixes for findings in pull requests and on the default branch. When Code Quality is enabled, two types of analysis run: CodeQL quality queries identify problems with the maintainability, reliability, or style of code, and LLM-powered analysis provides additional insights beyond what deterministic engines can find. When a quality issue is detected, Copilot Autofix suggests a relevant fix. On pull requests, results are displayed as comments left by the `github-code-quality` bot. On the default branch, LLM-powered findings are displayed in the **AI findings** dashboard under the **{% octicon "shield" aria-hidden="true" aria-label="shield" %} {% ifversion security-and-quality-tab %}Security and quality{% else %}Security{% endif %}** tab.
+GitHub Code Quality helps users improve code reliability, maintainability, and overall project health by surfacing actionable feedback and offering automatic fixes for findings in pull requests and on the default branch. When Code Quality is enabled, two types of analysis run: CodeQL quality queries identify problems with the maintainability, reliability, or style of code, and LLM-powered analysis provides additional insights beyond what deterministic engines can find. When a quality issue is detected, Copilot Autofix suggests a relevant fix. On pull requests, results are displayed as comments left by the `github-code-quality` bot. On the default branch, LLM-powered findings are displayed in the **AI findings** dashboard under the **{% octicon "shield" aria-hidden="true" aria-label="shield" %} {% ifversion security-and-quality-tab %}Security and quality{% else %}Security{% endif %}** tab. Additionally, Code Quality can automatically generate a code coverage CI workflow for your repository using an AI-powered agent, reducing the manual effort required to set up coverage tracking.
 
 The primary supported language for GitHub Code Security AI features is English.
 

data/reusables/code-quality/workflow-generation.md

@@ -0,0 +1 @@
+You can use the automatic setup option to generate a workflow with an AI-powered agent, or configure your workflow manually. See [AUTOTITLE](/code-security/how-tos/maintain-quality-code/set-up-code-coverage).