Separate Unit and Integration Test Runs #3907

paulmedynski · 2026-01-21T15:58:00Z

Motivation

Our PR pipelines are taking too long to run, for many reasons that we can't fix all at once. The low hanging fruit right now is that we are running all MDS tests twice - once in the Project pipeline, and once in the Package pipeline. This is unnecessary, so we're splitting things up:

Project runs will perform unit tests (i.e. MDS Unit and Functional suites).
Package runs will perform integration tests (i.e. MDS Manual suite).

Reasoning

The MDS Unit test suite requires Project reference builds anyway, so it makes sense to keep it in that pipeline.
The MDS Functional tests are (in theory) unit tests in disguise, and they don't typically take very long to run, so lumping them in with the Unit test suite seems fine.
The Manual tests take a long time, and they are intended to be more black-box, hence Package-based builds make sense for them.
The other pipelines (CI, downstream in ADO.Net, etc) will continue to run all tests multiple times).

Testing

PR pipeline runs will confirm that tests are still passing. I will manually verify that the expected types of tests are running in the designated pipelines.

Copilot

Pull request overview

This pull request optimizes CI/CD pipeline execution by separating unit and integration tests to reduce redundant test execution. The PR introduces a testType parameter that controls which test suites run in different pipeline contexts.

Changes:

Added testType parameter ('All', 'Unit', or 'Integration') to pipeline templates
Modified PR pipelines to run only relevant test suites (Unit tests in Project builds, Integration tests in Package builds)
Updated test execution conditionals to respect the new testType parameter

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
eng/pipelines/sqlclient-pr-project-ref-pipeline.yml	Sets testType to 'Unit' for project reference builds, restricting to unit and functional tests
eng/pipelines/sqlclient-pr-package-ref-pipeline.yml	Sets testType to 'Integration' for package reference builds, restricting to manual tests
eng/pipelines/dotnet-sqlclient-ci-core.yml	Defines testType parameter with default 'All' for CI pipelines to continue running all tests
eng/pipelines/common/templates/steps/run-all-tests-step.yml	Implements conditional test execution based on testType parameter
eng/pipelines/common/templates/stages/ci-run-tests-stage.yml	Propagates testType parameter through stage template
eng/pipelines/common/templates/jobs/ci-run-tests-job.yml	Propagates testType parameter through job template

eng/pipelines/common/templates/stages/ci-run-tests-stage.yml

eng/pipelines/common/templates/steps/run-all-tests-step.yml

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml

codecov · 2026-01-21T17:06:00Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.81%. Comparing base (0ec7af1) to head (523250b).

❗ There is a different number of reports uploaded between BASE (0ec7af1) and HEAD (523250b). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (0ec7af1) HEAD (523250b)

addons 1 0

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3907       +/-   ##
===========================================
- Coverage   74.99%   28.81%   -46.19%     
===========================================
  Files         269      263        -6     
  Lines       43246    66170    +22924     
===========================================
- Hits        32433    19064    -13369     
- Misses      10813    47106    +36293

Flag	Coverage Δ
addons	`?`
netcore	`28.01% <ø> (-47.05%)`	⬇️
netfx	`29.94% <ø> (-44.23%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

benrr101

Overall, probably a good change. The main issue I have right now is the lumping of functional tests with unit tests. My understanding of them were that they were not unit tests in theory, they were just the simple integration tests that validate they do the most basic stuff. As it turns out a lot of those are unit tests, but not the majority. I was working on the side a bit to move the unit tests from the functional tests and into the unit test project. Ultimately I think we're aligned that we want: unit test, local integration test, remote integration test, stress test, and fuzz test projects. And this change is kinda orthogonal to those goals, without getting in the way of them. So go for it 😆

I think another low-ish hanging fruit is to separate the unit tests and functional tests as separate jobs, like the test sets. That way we don't need to repeat unit and functional tests four times for each platform. They're fast, yes, but not instantaneous, and I imagine we could shave a decent amount of time off a build doing it.

paulmedynski · 2026-01-21T18:05:22Z

You're correct on all counts. I chose to combine the running of Unit and Functional tests because they're mostly unit with some local integration sprinkled in, but mainly because their combined run time is close to the Manual ones. This gives us the most bang-for-the-buck in terms of speeding up the PR runs right now.

We also shouldn't be running Unit and Functional tests in any of the Azure SQL configurations - another future optimization.

paulmedynski · 2026-01-21T19:36:08Z

eng/pipelines/dotnet-sqlclient-ci-core.yml


-      # Include the code coverage job if the build type is Project.
-      ${{ if eq(parameters.buildType, 'Project') }}:
+      # Include the code coverage job under certain circumstances:


This will continue to publish code coverage the same as we did before the split... for now. This is fragile and we should re-think how/when/why to publish coverage as we re-write these pipelines.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

eng/pipelines/dotnet-sqlclient-ci-core.yml

priyankatiwari08

Looks good to me. Manual tests in itself takes quite significant amount of time to run. So, this segregation should help us in longer run.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml

- Project runs will perform unit tests (i.e. MDS Unit and Functional suites). - Package runs will perform integration tests (i.e. MDS Manual suite).

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

- Use a temp dir to avoid cluttering up the pooled agents. - Combine all merging/converting into a few files as possible. - Use the modern features of the dotnet tools to avoid parallel jobs and loops. - Combined all MDS reports into a single upload, and same for AKV.

paulmedynski · 2026-01-23T20:59:57Z

eng/pipelines/dotnet-sqlclient-ci-core.yml

        - template: common/templates/jobs/ci-code-coverage-job.yml@self
          parameters:
-            debug: ${{ parameters.debug }}
+            # TODO: Forcing debug temporarily for PR runs.


@paulmedynski Address this TODO.

paulmedynski · 2026-01-23T21:00:33Z

eng/pipelines/dotnet-sqlclient-ci-core.yml

-            # buildType don't add much, if any, value to the coverage reports.
-            # They have not been configured with suitable CodeCov credentials to
-            # perform the upload.
+            # Other pipelines that share this template buildType don't add much,


@paulmedynski Fix this jibberish.

paulmedynski · 2026-01-23T21:01:44Z

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

    variables:
-      netFxDir: $(Build.SourcesDirectory)\coverageNetFx
-      netCoreDir: $(Build.SourcesDirectory)\coverageNetCore
+      # Use a temp directory that is cleaned up after each job runs.  This helps


I will run the pipeline in debug mode to make sure this temp dir lives on a partition/mount with sufficient space.

paulmedynski · 2026-01-23T21:03:52Z

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

-            ${{ else }}:
-              targetPath: $(netCoreDir)
+
+      # Download all of the coverage reports from the test jobs.


I greatly simplified things here. The old tasks went out of their way to separate the MDS .NET and .NET Framework coverage and reports. Now that we've merged the codebases, I can't see why we would care about that. So the new approach is to merge all of the test results together, generate Cobertura reports for MDS and AKV, and then upload them as separate entities. Thoughts?

paulmedynski requested a review from a team as a code owner January 21, 2026 15:58

Copilot AI review requested due to automatic review settings January 21, 2026 15:58

paulmedynski added Area\Tests Issues that are targeted to tests or test projects Area\Engineering Use this for issues that are targeted for changes in the 'eng' folder or build systems. labels Jan 21, 2026

Copilot started reviewing on behalf of paulmedynski January 21, 2026 15:58 View session

Copilot AI reviewed Jan 21, 2026

View reviewed changes

eng/pipelines/common/templates/stages/ci-run-tests-stage.yml Show resolved Hide resolved

eng/pipelines/common/templates/steps/run-all-tests-step.yml Show resolved Hide resolved

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml Show resolved Hide resolved

benrr101 previously approved these changes Jan 21, 2026

View reviewed changes

paulmedynski dismissed benrr101’s stale review via 546a0eb January 21, 2026 18:20

paulmedynski commented Jan 21, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings January 22, 2026 00:29

Copilot started reviewing on behalf of paulmedynski January 22, 2026 00:29 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

eng/pipelines/dotnet-sqlclient-ci-core.yml Outdated Show resolved Hide resolved

eng/pipelines/dotnet-sqlclient-ci-core.yml Outdated Show resolved Hide resolved

priyankatiwari08 self-assigned this Jan 22, 2026

priyankatiwari08 previously approved these changes Jan 22, 2026

View reviewed changes

paulmedynski dismissed priyankatiwari08’s stale review via e170d4d January 22, 2026 18:35

Copilot AI review requested due to automatic review settings January 22, 2026 18:46

Copilot started reviewing on behalf of paulmedynski January 22, 2026 18:46 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml Outdated Show resolved Hide resolved

eng/pipelines/sqlclient-pr-project-ref-pipeline.yml Outdated Show resolved Hide resolved

apoorvdeshmukh previously approved these changes Jan 23, 2026

View reviewed changes

PR pipeline runs will separate which tests they run:

523250b

- Project runs will perform unit tests (i.e. MDS Unit and Functional suites). - Package runs will perform integration tests (i.e. MDS Manual suite).

Copilot AI review requested due to automatic review settings January 23, 2026 13:52

paulmedynski dismissed apoorvdeshmukh’s stale review via 523250b January 23, 2026 13:52

paulmedynski force-pushed the dev/paul/duplicate-runs branch from 190fcb7 to 523250b Compare January 23, 2026 13:52

Copilot started reviewing on behalf of paulmedynski January 23, 2026 13:53 View session

Copilot AI reviewed Jan 23, 2026

View reviewed changes

paulmedynski commented Jan 23, 2026

View reviewed changes

Separate Unit and Integration Test Runs #3907

Are you sure you want to change the base?

Separate Unit and Integration Test Runs #3907

Conversation

paulmedynski commented Jan 21, 2026

Motivation

Reasoning

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

benrr101 left a comment

Choose a reason for hiding this comment

Uh oh!

paulmedynski commented Jan 21, 2026

Uh oh!

paulmedynski Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

priyankatiwari08 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

paulmedynski Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

paulmedynski Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

paulmedynski Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

paulmedynski Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Jan 21, 2026 •

edited

Loading